Cloud Bursting from On-Premise Kubernetes Clusters to Google Cloud Compute Engine
What is Cloud Bursting?
The capability to handle unplanned peak demands or planned surges in traffic by using Public clouds to add extra capacity on-demand can be very appealing to Hybrid cloud operators. This concept of deployment, known as Cloud Bursting, is ideal for batch jobs which are not latency sensitive and it is a cost effective solution which brings a lot of flexibility when private data center capacity is strained.
Cloud Bursting is useful in many production Hybrid cloud environments to help solve short term capacity constraints in the Private data centers by bursting into Public clouds when computing demands spike. For example, an e-commerce application hosted on a Private cloud may need that extra capacity during the holiday season to handle peak load due to an anticipated increase in online shopper traffic. Computing resources can be provisioned on-demand in the Public cloud to handle the peak load, and they can be released after the holiday season. With this capability, Hybrid cloud operators can now reduce the heavy capital cost of owning and managing excess compute capacity and storage in their Private data centers.
Virtual Kubelet
Virtual Kubelet is an Open Source project under the Cloud Native Computing Foundation (CNCF)’s sandbox project. Virtual Kubelet masquerades as a Kubernetes Kubelet that extends as a virtual worker node in your on-premise Kubernetes cluster. This virtual worker node can be backed by public cloud services such as Google Compute Engine (GCE). When a container pod is scheduled onto this virtual worker node, Virtual Kubelet will quickly provision a GCE instance on-demand to run the new container pod. Today, most on-premise enterprise Kubernetes platforms do not implement nor support the capabilities of Cluster Autoscaler to dynamically add or remove worker nodes or node pools to match current cluster utilization. As such, manually adding additional compute worker nodes to an on-premise Kubernetes cluster cannot match the speed, scale and flexibility of Cloud Bursting to Public cloud.
The Virtual Kubelet project provides a software library which is implemented by several different entities known as providers. Each provider performs the backend configuration necessary to manage the on-demand resources required to run the container pods scheduled onto the virtual nodes registered into the Kubernetes cluster. In this article, I will talk about using Elotl’s Kip, a Virtual Kubelet provider which allows a Kubernetes cluster to transparently launch container pods onto GCE instances running in a Google Cloud region closest to your on-premise data center.
How to burst into Google Cloud Platform?
The diagram below shows the Hybrid cloud environment I will be using to illustrate Cloud Bursting from an on-premise Kubernetes cluster into Google Cloud GCE instances. The Hybrid cloud environment requires private network connectivity between the on-premise data center and a Google Cloud region. This private connection can be fulfilled with a direct dedicated interconnect to a Google Cloud region. Alternatively, IPSEC based Cloud VPN over the Internet or a Software Defined WAN (SD-WAN) from 3rd party vendors over the Internet can be used in the event that the on-premise data center location does not have the dedicated interconnect option available.
Clone the Kip Github project
The first step is to clone the Kip Github project at https://github.com/elotl/kip
Creating the GCP service account and credentials
Next, we need to create a GCP service account required by the Kip virtual kubelet agent. An example is shown below. You can use any name you like for the service account name.
As shown in the next example below, we will bind this service account with the roles required (iam.serviceAccountUser and compute.admin) by the Kip provider. The Kip provider agent requires the role to use this service account as a User as well as to create and delete GCE instances as a compute admin.
Once the required roles are associated with the service account, we will now download the service account JSON key.
Open up the JSON key file downloaded onto your machine and look for the “private key” field in the file which we will need to populate in the provider.yaml file in the next section. Edit the provider.yaml file in your cloned Kip directory and populate the relevant information inside the file required by the Kip Virtual Kubelet provider. As shown in the provider.yaml config example shown below, for Cloud Bursting to Google Cloud’s GCE, you have to provide a few pieces of information which include GCP project ID, the email of the GCP service account which has compute.admin and serviceAccount.user roles associated with it, the Private Key information in the JSON key downloaded from your GCP service account, the Virtual Private Cloud (VPC) name, the GCP region’s availability zone and subnetwork name where the GCE instances will be provisioned in. Once all these information is filled in, save the provider.yaml file.
Applying the Kip manifest to your Kubernetes cluster
We will use kustomize configuration template tool to create the Kubernetes YAML manifests including the provider.yaml file, and apply the manifests to your on-premise Kubernetes cluster.
The Virtual Kubelet Kip provider will create a Persistent Volume Claim which is bound to a Persistent Volume to store stateful transactional data of GCE instance creations and deletions etc.
A Kubernetes Statefulset is created on the Kubernetes cluster which will spawn two kip-provider multi-container pods and a virtual node will be registered to the cluster as a worker node.
Scheduling pods onto the Virtual Kubelet worker node
Next, we’re ready to schedule container pods onto the Virtual Kubelet worker node. A simple Nginx deployment manifest is used whereby the nodeSelector field is set to “type: virtual-kubelet” in the manifest to explicitly schedule the container pods onto the virtual worker node. Under normal circumstances, we can let the kube-scheduler find the feasible worker nodes to schedule container pods.
Validate the GCE instance is provisioned by Virtual Kubelet carrying out Cloud Bursting
As soon as the earlier Nginx deployment manifest is applied, the kube-scheduler selects the virtual worker node and we can examine the kip container inside the kip-provider container pod to take a closer look at what is happening. As shown in the example below of the kip container logs, the kip container invokes a new GCE instance creation once the Nginx container pod is scheduled onto the virtual worker node.
A new GCE instance is created on-demand by the kip container in the VPC network and GCP availability zone specified much earlier in the provider.yaml file.
The container image is downloaded and a new container pod is launched remotely onto the GCE instance. The Kubernetes cluster’s kubeconfig file is passed to the kube-router agent running on the GCE instance, whereby the kube-router agent runs the container image and creates a layer 3 connection back to the on-premise Kubernetes cluster’s kube-api server over the dedicated interconnect or Cloud VPN connection. The GCE instance is extremely lightweight as it does not require a container run time engine. It is also worthwhile to highlight the etcd database remains secured running in the Kubernetes cluster on-premise.
After a few seconds, the new Nginx container pod is now in the running state on the GCE instance.
Validate Ingress traffic flow from clients to container pods on virtual worker node
The Nginx deployment can be exposed as an external service using an Ingress controller, where traffic from clients to the container pod will enter the Kubernetes cluster via the on-premise Load Balancer and data center networking. Once client traffic hits the service, it will route over the private dedicated interconnect or Cloud VPN to the Nginx container endpoints running remotely on the GCE instances on the Public cloud side.
Transparent to the clients, the service is accessible via Ingress to the on-premise Kubernetes cluster but the container pods are Cloud Bursted to the Public cloud.
Summary
Cloud Bursting provides a solution to meet seasonal demands by leveraging on Public cloud resources during times of increased workload and traffic without incurring excess capital expenditure. When traffic returns to normal levels, data can be moved back to the Private cloud. Cloud Bursting is also generally recommended for high performance and stateless workloads that handle nonsensitive information.
