written by: Vlad

NextGen Kubernetes Infrastructure

Deploying multi-domain SaaS onto K8S

Blugento is a Software-as-a-Service built on top of Magento, and its business is to develop and host thousands of shops, each with its own domain, data and customisation.

Initially, Blugento ran on a VM based setup, multi-VM with load balancer for the larger stores. However, as Blugento quickly grew to exceed 200 shops, it became clear that this model is not scalable.

Traditional VM Hosting

Beside the extremely high cost, additional downsides of this setup was that orchestration was difficult and some shops didn’t have redundancy in their hosting.

To solve this problem, a container-based hosting approach was the obvious solution. We first looked at Docker, however a setup with solutions like Amazon Container Service wasn’t appealing mainly because it was too platform specific.

Kubernetes however was the new hot thing, so we decided to go for it. The end result looks like the diagram below, however the road to it wasn’t easy.

Blugento Kubernetes Setup

We the following goals for the new infrastructure:

  • Cost efficient: goodbye to 99% idle CPU/Memory like we had on the VMs
  • Scalable: ability to scale up and down automatically based on load in under 1min
  • Redundancy: all the components redundant across multiple availability zones
  • Platform agnostic: be able to run it on AWS, Azure or GCP
  • PaaS: use PaaS for database, caching and NFS storage

The first challenge was setting up the load balancer and multi-AZ database, caching and NFS storage. On AWS this consisted of ELB, Aurora RDS, ElastiCache and EFS. The user visit would hit an Elastic Load Balancer, which would route to the k8s load balancer (we started off with Voyager, but due to technical issues we ended up building our own k8s haproxy load balancer). The haproxy then routed the request to the frontend service, which contained at least 2 pods on different nodes (with container for nginx and php7); those pods would connect to RDS for database, ElastiCache for Redis and mount an EFS for the media and other dynamic folders.

Additional pods for Cron, Grunt, and also daily jobs for backup were relatively easy to add. Inside k8s we’re also running all the other pods for load balancing, monitoring, mailing services etc. in the same fully scalable and stable approach.
The next challenge was initialising the pods with the source codes; we knew from the start that we couldn’t keep the Magento files in EFS due to performance reasons, so each pod needed to host its own sources on fast SSD drives. We looked at pulling the sources from Git, however the speed or pulling 20+ modules delayed the boot too much; BitBucket availability was also a concern. The solution was securely downloading the sources from S3, where they are deployed through our CI processes.

Final challenge was the orchestration, where we built a custom system to track all the shops with their modules, configurations and scaling rules. Now updating a store is just a couple of clicks, with commands feeding directly into the Kubernetes API. Error logging, uptime and performance monitoring are also automated inside this solution.

The result is a very stable and scalable architecture, with ultra-fast deployment and 10x cost optimisation.

Although the Blugento deployment is quite unique due to the SaaS architecture, a similar high performance deployment can also be achieved for a single application, especially for those with a micro-service architecture.

Interested? Get in touch

Other articles Comments off

Comments are closed.