This is a guest article by technical writer Limor Maayan-Wainstein
Kubernetes is a popular open source container orchestration platform used by many organizations worldwide. Kubernetes is feature-rich, providing the controls needed for managing the deployment of enterprise-grade applications. However, Kubernetes is also decidedly complex and difficult to learn and operate. These Kubernetes complexities can lead to misconfigurations and create vulnerabilities in your entire ecosystem.
According to recent research, 94 percent of those surveyed had experienced a container security incident in the last year. Due to these incidents, 44 percent delayed moving workloads to production, affecting productivity and revenue.
Kubernetes complexity
Although containers benefit from isolation, which can increase security, Kubernetes networking presents complexity that is difficult to secure. Exposing hundreds or thousands of services both internally and externally leaves many entry points for attackers and can reduce visibility. Deploying and interconnecting the various moving pieces of a deployment leaves a lot of room for human error and oversight.
Kubernetes misconfiguration
Misconfigurations can leave both data and systems vulnerable, allowing abuse of permissions, resources, and theft of data. According to the same survey cited above, 61 percent surveyed reported misconfigurations as their greatest concern. This was compared to 27 percent who were worried about vulnerabilities and 12 percent who worried about attacks.
Kubernetes skills
Another issue is preparedness, or the lack of, that teams have. Attempting to operate Kubernetes in production with a traditional team rather than a DevOps one, or transitioning to DevOps to use Kubernetes are ill-advised. Additionally, while it’s possible to learn Kubernetes from scratch, immediately deploying to production without thorough testing and verification of new skills is rash.
For example, ensuring that you have pod security policies defined can help you secure your multi-tenant clusters. However, depending on your version, this feature may be in beta (requiring activation) or may not be available. Understanding which built-in features are available to you helps you understand where you need to supplement your security.
When reviewing your configurations, use available security benchmarks, such as those created by the Center for Internet Security (CIS). These benchmarks provide a guide to help you harden your systems and identify vulnerabilities. Benchmarks can also help ensure you meet compliance with your deployment, providing proof of audits and measures taken.
Another consideration is how well integrated security is with your daily operations. Be careful you don't let security unnecessarily impede workflows.
One solution for this is the integration of security into your DevOps team, moving to DevSecOps. Integrating your teams enables you to start from a secure base rather than tacking on security. This enables teams to develop workflows that flow smoothly despite or even because of security measures. It also reduces the chance that vulnerabilities make it to production.
DevSecOps teams can also enable you to better leverage your existing automation, improving some of the auditing or monitoring tasks that were previously manual or disorganized, for example, log aggregation or alerting.
Also, with teams working together, unification tools, such as service meshes, are potentially easier to implement and can provide greater benefit. This is because teams can collaborate on needs and concerns, resulting in fewer roadblocks or misunderstandings during configuration.
For example, Helm, which is used to leverage existing machine images and configurations, is great for provisioning initial resources and deploying applications but it doesn’t provide much in the way of ongoing operations. For this, you need Kubernetes operators.
Operators enable you to package, deploy, and manage your applications by encoding operational directions and bundling these directions with your applications. You can use operators to automate lifecycle management and ensure standardized deployments. For day 2 security, you should also use them to ensure that only supported configurations are allowed to continue running.
Another day 2 concern that may be overlooked is leftover environments. Getting to production requires significant testing and practice configurations that can result in many insecure resources and configurations along the way.
Part of your day 2 responsibilities include making sure that these resources are killed and that trial configurations are unavailable for future deployments. Responsibilities also include ensuring that any ports or permissions used for initial testing are properly disabled and that temporary permissions are removed.
When setting up RBAC, you should prioritize namespace-specific permissions over cluster-wide permissions for more granular control. You should also avoid including cluster admin privileges in any role. These permissions are not often needed and can be more securely managed with manual access granting.
Make sure that you have TLS enabled for both client-to-server and server-to-server communications to prevent request and response data from being compromised. You should also place a firewall to filter traffic and provide visibility into any data transfers that occur. Place this firewall between etcd and your API server and consider running etcd on separate nodes for greater security.
To isolate your streams, configure your nodes to only allow master node connections on specific ports. This can be accomplished with access control lists (ACLs) and significantly reduces the traffic you need to monitor. It also makes it easier to intercept harmful connections.
Whitelisting does not require you to know anything about whether unknown processes are malicious or not. This means you can effectively block zero-day or dynamic attack processes.
If you have one service that is operating differently from others, it may be an indication that something is wrong. Comparing the behavior of your resources and services can enable you to determine if a pod has been compromised or deployed illegitimately.
Kubernetes provides native support for monitoring through Prometheus but this is primarily for metrics. For security purposes, you should consider additional utilities such as service meshes, which can aggregate visibility factors and enable greater control.
Kubernetes security considerations include configuring for hardening and compliance, enforcing a balance between security and agility, and monitoring security after deployments. Continuous monitoring can help you achieve visibility at all times, ensuring that your environment remains secure, and that you can respond quickly when security issues arise.
Limor Maayan-Wainstein is a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more. She is the winner of the STC Cross-European Technical Communication Award (2008) and a regular contributor to technology publications.
Want to write an article for our blog? Read our requirements and guidelines to become a contributor.
Kubernetes is a popular open source container orchestration platform used by many organizations worldwide. Kubernetes is feature-rich, providing the controls needed for managing the deployment of enterprise-grade applications. However, Kubernetes is also decidedly complex and difficult to learn and operate. These Kubernetes complexities can lead to misconfigurations and create vulnerabilities in your entire ecosystem.
Kubernetes Security Concerns in the Field
Kubernetes is known for both its effectiveness and its complexity. Orchestrating container deployments, especially at scale, can be a significant challenge. A large part of this challenge is ensuring secure deployments.According to recent research, 94 percent of those surveyed had experienced a container security incident in the last year. Due to these incidents, 44 percent delayed moving workloads to production, affecting productivity and revenue.
Kubernetes complexity
Although containers benefit from isolation, which can increase security, Kubernetes networking presents complexity that is difficult to secure. Exposing hundreds or thousands of services both internally and externally leaves many entry points for attackers and can reduce visibility. Deploying and interconnecting the various moving pieces of a deployment leaves a lot of room for human error and oversight.
Kubernetes misconfiguration
Misconfigurations can leave both data and systems vulnerable, allowing abuse of permissions, resources, and theft of data. According to the same survey cited above, 61 percent surveyed reported misconfigurations as their greatest concern. This was compared to 27 percent who were worried about vulnerabilities and 12 percent who worried about attacks.
Kubernetes skills
Another issue is preparedness, or the lack of, that teams have. Attempting to operate Kubernetes in production with a traditional team rather than a DevOps one, or transitioning to DevOps to use Kubernetes are ill-advised. Additionally, while it’s possible to learn Kubernetes from scratch, immediately deploying to production without thorough testing and verification of new skills is rash.
Kubernetes Security: Key Considerations
When deploying Kubernetes, there are a few key considerations every team should keep in mind. These considerations can help you ensure the basics are covered without impeding productivity.Hardening and compliance
Properly securing Kubernetes requires focus on what features you have enabled and which you haven't. This includes features that are enabled or disabled by default.For example, ensuring that you have pod security policies defined can help you secure your multi-tenant clusters. However, depending on your version, this feature may be in beta (requiring activation) or may not be available. Understanding which built-in features are available to you helps you understand where you need to supplement your security.
When reviewing your configurations, use available security benchmarks, such as those created by the Center for Internet Security (CIS). These benchmarks provide a guide to help you harden your systems and identify vulnerabilities. Benchmarks can also help ensure you meet compliance with your deployment, providing proof of audits and measures taken.
Balancing security and agility
While security is essential, it is unhelpful to layer on so much security that your deployment becomes unfunctional. Too many authentication processes or too heavily restricted permission stand in the way of productivity and can negate any benefit Kubernetes may provide.Another consideration is how well integrated security is with your daily operations. Be careful you don't let security unnecessarily impede workflows.
One solution for this is the integration of security into your DevOps team, moving to DevSecOps. Integrating your teams enables you to start from a secure base rather than tacking on security. This enables teams to develop workflows that flow smoothly despite or even because of security measures. It also reduces the chance that vulnerabilities make it to production.
DevSecOps teams can also enable you to better leverage your existing automation, improving some of the auditing or monitoring tasks that were previously manual or disorganized, for example, log aggregation or alerting.
Also, with teams working together, unification tools, such as service meshes, are potentially easier to implement and can provide greater benefit. This is because teams can collaborate on needs and concerns, resulting in fewer roadblocks or misunderstandings during configuration.
Day 1 vs day 2 operations
An area that is often overlooked is what happens on day 2. For many organizations, all of the planning and effort is focused on day 1 operations, getting Kubernetes up and running successfully. This often involves a range of utilities and tools to help the process along.For example, Helm, which is used to leverage existing machine images and configurations, is great for provisioning initial resources and deploying applications but it doesn’t provide much in the way of ongoing operations. For this, you need Kubernetes operators.
Operators enable you to package, deploy, and manage your applications by encoding operational directions and bundling these directions with your applications. You can use operators to automate lifecycle management and ensure standardized deployments. For day 2 security, you should also use them to ensure that only supported configurations are allowed to continue running.
Another day 2 concern that may be overlooked is leftover environments. Getting to production requires significant testing and practice configurations that can result in many insecure resources and configurations along the way.
Part of your day 2 responsibilities include making sure that these resources are killed and that trial configurations are unavailable for future deployments. Responsibilities also include ensuring that any ports or permissions used for initial testing are properly disabled and that temporary permissions are removed.
Kubernetes Security Best Practices
When configuring and verifying your Kubernetes security, there are several best practices you should apply. During your periodic audits, ensure that these practices remain in place and modify configurations as needed.Enable role-based access controls (RBACs)
RBAC is enabled by default in most Kubernetes versions and is recommended for easier and more secure management of permissions. With RBAC you can quickly update permissions and easily predict what users or resources can do. This enables you to standardize controls and reduces the chance that excess permissions are granted.When setting up RBAC, you should prioritize namespace-specific permissions over cluster-wide permissions for more granular control. You should also avoid including cluster admin privileges in any role. These permissions are not often needed and can be more securely managed with manual access granting.
Protect your etcd cluster
Your etcd cluster is one of the most valuable targets for attackers so properly protecting it is key. If an attacker can gain read access to your cluster, they can see all of your configurations, deployment status, and secrets. If they gain write access, they can take over your deployment.Make sure that you have TLS enabled for both client-to-server and server-to-server communications to prevent request and response data from being compromised. You should also place a firewall to filter traffic and provide visibility into any data transfers that occur. Place this firewall between etcd and your API server and consider running etcd on separate nodes for greater security.
Isolate Kubernetes nodes
You need to isolate your nodes as much as possible. This means ensuring that nodes are not directly exposed to public networks and ideally, not even to corporate networks. This requires isolating your data and controlling traffic streams. Without isolation, access to one provides access to the other.To isolate your streams, configure your nodes to only allow master node connections on specific ports. This can be accomplished with access control lists (ACLs) and significantly reduces the traffic you need to monitor. It also makes it easier to intercept harmful connections.
Leverage process whitelisting
Process whitelisting enables you to restrict processes to only those that are known and approved. This is one of the most effective ways to ensure that malicious processes are blocked.Whitelisting does not require you to know anything about whether unknown processes are malicious or not. This means you can effectively block zero-day or dynamic attack processes.
Compare pod runtime activities
Kubernetes deployments depend on replication for scaling, fault tolerance, availability, and automation. This means that many of your runtime activities should look decidedly similar or identical.If you have one service that is operating differently from others, it may be an indication that something is wrong. Comparing the behavior of your resources and services can enable you to determine if a pod has been compromised or deployed illegitimately.
Monitor your deployments
As with any system, continuous monitoring is vital to ensuring that your deployment remains secure and performant. This includes monitoring all connections, requests, processes, and any configuration changes. It also involves responding to any alerts as soon as possible to ensure that potential for damage is limited.Kubernetes provides native support for monitoring through Prometheus but this is primarily for metrics. For security purposes, you should consider additional utilities such as service meshes, which can aggregate visibility factors and enable greater control.
Conclusion
Kubernetes is highly popular but it takes time and skill to master this platform. Without mastery, you might experience a high number of vulnerabilities. When deploying Kubernetes, there are a few key considerations you should be certain of to ensure the continuous health of your pipeline.Kubernetes security considerations include configuring for hardening and compliance, enforcing a balance between security and agility, and monitoring security after deployments. Continuous monitoring can help you achieve visibility at all times, ensuring that your environment remains secure, and that you can respond quickly when security issues arise.
Limor Maayan-Wainstein is a senior technical writer with 10 years of experience writing about cybersecurity, big data, cloud computing, web development, and more. She is the winner of the STC Cross-European Technical Communication Award (2008) and a regular contributor to technology publications.
Want to write an article for our blog? Read our requirements and guidelines to become a contributor.