Detecting Threats and Anomalies in your Kubernetes Clusters in Multicloud environments

Threats Surveillance and Detection in the Multicloud

Jonathan Chin
4 min readJun 15, 2020

As we modernize our legacy applications into microservices and deploy them to container technologies such as Kubernetes, potential threats and unknown security risks lie in wait for us in the new environments. For example, the Common Vulnerabilities and Exposure (CVE) publishes the list of known vulnerabilities and exploits found in base Docker container images which we commonly use as base images to build our new containerized applications.

https://www.cvedetails.com/product/28125/Docker-Docker.html?vendor_id=13534

Some of these exploits are nasty, for example they allow hackers to execute arbitrary code with root privileges. Once the perpetrators have escalated privileges to access one of your containers, they can use it to launch other attacks from within your Kubernetes Cluster.

A case in point, recently it was reported that hackers gained illegal access to high value Kubernetes Clusters on one of the public cloud providers. These Kubernetes Clusters were built for Machine Learning applications with GPU resources and they were exploited by hackers to run cryptocurrency mining jobs.

https://www.sdxcentral.com/articles/news/hackers-cryptojack-microsoft-azure-ml-clusters/2020/06/

Securing Kubernetes Clusters

There are well documented best practices on securing your Kubernetes Clusters and protecting them from accidental or malicious access. Best practices such as securing your Kubernetes API server, restricting network access, and encrypting your secrets are necessary, but these alone may not make your Kubernetes environment foolproof and security threats may still escape detection.

To further protect your Kubernetes environment on the Multicloud, advanced real-time cloud security which can accurately identify compromised and misused resources becomes essential.

Stealthwatch Cloud

Stealthwatch Cloud is a Software as a Service (SaaS) based network visibility and security analytics solution that can protect any Kubernetes clusters in any cloud environment. It works on managed Kubernetes service offered by the major Cloud Providers (AWS EKS, Azure AKS, Google GKE) and also on on-premises Kubernetes Clusters such as Cisco Container Platform and Red Hat Openshift Container Platform.

Stealthwatch Cloud uses a Kubernetes daemonset configuration to deploy the lightweight sensors onto your Kubernetes Cluster. The Kubernetes daemonset ensures a sensor container pod will be deployed on every Kubernetes Node in your Cluster, including the Master Node. When the Kubernetes Cluster is scaled up, a new container pod will be automatically deployed on it.

Example of Kubernetes Daemonset YAML for Sensor deployment

Deploying the sensors on your Kubernetes Clusters running on the public cloud providers is easy. The daemonset configuration based on the earlier example is created on the Kubernetes Cluster, and the sensor container Pods will be created on the Kubernetes Nodes.

These sensors monitor, baseline and detect security anomalies within your Kubernetes Clusters. Container to container traffic, pod to pod and node to node traffic is monitored by the deployed sensors.

Detecting Threats and receiving Alerts

When the sensors have detected security threats or anomalies in your Kubernetes Cluster, an alert is triggered within minutes and the administrator has many different options to receive the notifications.

Kubernetes administrators can subscribe to receive alerts within minutes on zero day threats, application exploits, and breach indicators of compromise.

These alerts can be tagged and then assigned to different users or teams within the organization to follow up on the investigations.

In the example below, the administrator is alerted about a never-seen-before container Pod being created on a Red Hat Openshift Cluster.

Stealthwatch Cloud detected a never-seen-before Container Pod on an Openshift Cluster

In another example, the administrator is alerted about a malicious port scan being performed from a container Pod within the Kubernetes Cluster.

Por scan attempts from a Container Pod within a Kubernetes Cluster

The Stealthwatch Cloud makes it easy to integrate with Cloud Native services. For example, it is possible to deliver the alerts as messages to cloud storage such as AWS S3 or GCP Storage.

Stealthwatch Cloud can help our journey to Multicloud safer by ensuring our security is always one step ahead of tomorrow’s attacks.

--

--

Jonathan Chin

Jonathan is an App Modernization Customer Engineer at Google Cloud, helping customers in their journey towards Cloud Native. He lives in Singapore.