Shift-Left Testing Applied to Kubernetes
The Kubernetes container orchestration engine is a complicated piece of software that offers many opportunities for misconfigurations. For example, the Kubernetes Configuration Benchmark report analyzed over 100,000 Kubernetes workloads and found an average of 328 Kubernetes misconfigurations per cluster, a nightmare for the professionals in charge of the infrastructure. This is probably why cybersecurity firm Cyble found over 900,000 Kubernetes clusters exposed to the Internet, because of misconfigurations.
In the last few years, the software industry has been increasingly adopting the shift-left testing strategy, which advocates getting testing done earlier in a deployment pipeline lifecycle. Because of its declarative nature, Kubernetes is a great tool where a shift-left strategy can be implemented. In this article, we will explore the benefits of shift-right and shift-left strategies — by looking at open source solutions Gatekeeper and Datree — for spotting Kubernetes cluster misconfigurations.
Shift-Right Testing
Before discussing shift-left testing, let’s cover its opposite. Shift-right testing postpones the testing phase as far as possible, generally before deploying to or in production. This approach is what has been historically used in the software industry. It has the advantage of catching things that may not be detected in a staging environment, such as performance issues, failure tolerance, or user experience in the case of an application.
Open source software Gatekeeper acts as a bridge between the Kubernetes API server and OPA policies. Whenever a resource creation, update, or deletion request is sent to the Kubernetes cluster, Gatekeeper sits as a validating webhook and will check that submitted requests submitted are not in infraction with predefined OPA policies. Because Gatekeeper is doing its work at the cluster level, which can be done at the staging phase before deployment or directly in production, it can be used as part of a shift-right testing strategy.
Because of its declarative nature, Kubernetes is a great tool where a shift-left strategy can be implemented.
However, it is not always an ideal scenario. Shift-right testing means that the infrastructure team, which is generally on the far end of the shipping pipeline, will have to take care of misconfigurations. This can quickly be an overwhelming task, especially if there are many developers, which generally drastically outnumbers the infrastructure team count.
And with an average of 328 misconfigurations per Kubernetes cluster, things can quickly get out of control. This can lead to a drastic loss of velocity for the application development lifecycle and cause frustration among developers who don’t see their code going to production.
Shift-Left Testing
That’s where a shift-left testing approach can help. By moving the testing phase earlier, ideally at the development stage, every developer becomes a testing unit. Because the testing stage is happening at their level, they will be more involved in catching misconfigurations compared to when the responsibility is attributed to another team. This will increase the test coverage, enhance the application delivery velocity, and avoid developers’ frustration with seeing their code sitting in preproduction.
Open-source software Datree offers a CLI tool — which spots Kubernetes misconfigurations — that can be executed directly on developers’ laptops or as part of a CI/CD pipeline and can be used as part of a shift-left testing strategy. One of the significant challenges of moving the testing phase to the left is obviously to get the developer’s buy-in to do the testing and actually make sure that they have the knowledge. That is why engineering leadership must ensure that the topic is widely discussed with the team.
Organizing brainstorming sessions to come up with a test baseline that everybody is aware of and agrees with is a must. Providing developers-friendly tools for spotting misconfiguration will also increase the chance of having the team onboard. For example, Datree offers to integrate the automated checking directly into the code editor — with their VS plugin — while providing developer-friendly error messages that don’t require extensive Kubernetes administration knowledge.
Once a shift-left testing strategy is in place and the developer’s team is on board; make sure to track progress. Similarly to tracking the coverage test for an application codebase, do the same for your Kubernetes configuration files. Remember to bring the topic during the team stands up, celebrate the success, and discuss testing challenges to ensure that shift-left testing becomes a smooth part of the application lifecycle.
Both shift-right and shift-left strategies have pros and cons, but the shift-left one is emerging, which is generally where companies need to improve. Ultimately, having both in place can provide solid testing coverage.
In today’s fast shipping CI/CD pipelines, fixing a Kubernetes misconfiguration as early as possible in the shipping process can reduce the misconfiguration cost from $15,000 to $8. So don’t let your company infrastructure slip into this 640x zone!