Amazon EKS now supports Kubernetes version 1.27

Introduction

The Amazon Elastic Kubernetes Service ( Amazon EKS ) team is pleased to announce support for Kubernetes version 1.27 for Amazon EKS and Amazon EKS Distro . Amazon EKS Anywhere (release 0.16.0) also supports Kubernetes 1.27. The theme for this version was chosen to recognize the fact that the release was pretty chill. Hence, the fitting release name, Chill Vibes. In their official release announcement , the Kubernetes release team said the release was, “the first release that anyone can remember where we didn’t receive a single exception request after the enhancements freeze.”

Kubernetes 1.27 highlights

This post highlights the most significant removals, deprecations, and features in Kubernetes version 1.27, which could potentially affect your cluster and workloads. For starters, it’s important to note that there are several features and versions being deprecated in v1.27, including the k8s.gcr.io image registry freeze. This means that you may need to update your workflows and configurations to avoid any potential issues. Finally, there’s a great kubelet enhancement we’re all excited about that allows you to easily secure your container workloads running on a given node. For a complete list of changes and updates in Kubernetes version 1.27, check out the Kubernetes change log .

There were some interesting features that become generally available (GA) in v1.27. Below are a few enhancements that has our technical community excited about the v1.27 release. For a complete list, refer to all feature graduations and deprecations in Kubernetes v1.27 .

SeccompDefault graduates to stable

#2413 . SeccompDefault has graduated to stable and the feature gate is enabled by default. This feature gate helps you enhance the security posture of your workloads without the need for extensive custom seccomp profiles. The default seccomp profiles are designed to strike a balance between security and application compatibility. With this graduation, the RuntimeDefault seccomp profile will be used as the default for all workloads. To set the default seccomp profile for all containers running on a node, pass the –kubelet-extra-args “–seccomp-default” flag in your bootstrap script or launch template. This sets the RuntimeDefault seccomp profile as the default for all workloads by default, which is defined by the container runtime, instead of using the Unconfined (seccomp disabled) mode. By using this feature, you add an extra layer of protection against common system call vulnerabilities exploited by malicious actors.
Some workloads that currently run without seccomp may experience breakages when seccomp is enabled by default. To ensure compatibility and prevent potential issues, we highly recommend testing your workloads in a staging or test environment before enabling seccomp by default. If any compatibility issues arise, you have the option to either disable seccomp or create custom profiles for those specific workloads. Although the default profile may not be as restrictive as some custom profiles, it provides a baseline level of security without compromising application interoperability. Additionally, if you require more granular control over seccomp profiles and want to create and implement custom profiles for your workloads, you can explore the security-profiles-operator . This operator allows you to define and manage custom seccomp profiles in your cluster.
To learn more about using kubelet arguments in your cluster, see Customizing managed nodes with launch templates .

More fine-grained pod topology spread policies reached beta

In earlier versions of Kubernetes prior to v1.27, achieving a balanced pod spread across various domains (e.g., kubernetes.io/hostname) was a difficult task. In response to these hurdles, Kubernetes v1.27 unveils an advanced suite of pod topology spread policies. These features, described in the KEPs ( #3022 , #3094 , #3243 ) , are immediately available as they are enabled by default. They collectively offer a powerful toolset that bolsters the capacity to evenly distribute workloads, amplifies resilience, and simplifies the execution of rolling upgrades.
Leading the way, #3022 unveils the minDomains parameter, gives you the ability to determine the minimum number of domains your pods should occupy, thereby guaranteeing a balanced spread of workloads across the cluster. Second in line, #3094 introduces the nodeAffinityPolicy and nodeTaintPolicy parameters, which allow for an extra level of granularity in governing pod distribution according to node affinities and taints. This particular feature is linked with the NodeInclusionPolicyInPodTopologySpread gate, now enabled by default for better utilization. Lastly, #3243 implements the matchLabelKeys field in the topologySpreadConstraints of your pod’s specification, which permits the selection of pods for spreading calculations following a rolling upgrade.
To learn more, refer to Kubernetes 1.27: More fine-grained pod topology spread policies reached beta .

Raised default API query-per-second limits for kubelet

In versions prior to v1.27, the Amazon EKS kubelet had default values of 10 requests per second for kubeAPIQPS and a burst limit of 20 requests for kubeAPIBurst, which determined the rate at which the kubelet could handle incoming requests. With #1040 , the Amazon EKS optimized AMI has adopted new default values of 50 requests per second for kubeAPIQPS and a burst limit of 100 requests for kubeAPIBurst. These changes improve how quickly pods start running on new nodes when there is a sudden demand for additional resources.
By increasing the default values for the kubelet, these enhancements allow the kubelet to handle a higher volume of API queries per second, enabling better responsiveness and performance. When there is a sudden demand for Pods due to scaling requirements, the revised defaults ensure that the kubelet can efficiently manage the increased workload, resulting in faster pod startups and smoother cluster operations. These adjustments in the kubelet defaults within the Amazon EKS optimized AMI align with the ongoing efforts to optimize Kubernetes performance and provide an improved experience to the customers.

Removed API versions and features

Nowadays, it’s not uncommon for Kubernetes Application Programming Interface (API) versions and features to be deprecated or removed when a new version of Kubernetes is released. When this happens, it’s imperative that you update all manifests and controllers to the newer versions and features listed in this section before upgrading to version 1.27 . Below are the top call-outs in the v1.27 release. For a complete list, refer to all Deprecations and removals in Kubernetes v1.27 .

k8s.gcr.io image registry is frozen

Starting from Kubernetes v1.27, the Kubernetes project has frozen the k8s.gcr.io registry. This means that new Kubernetes images from v1.27 and patches for older versions won’t be published to the k8s.gcr.io registry. Instead, you must start using registry.k8s.io to get the latest Kubernetes images, which is now the only source for Kubernetes images. If you haven’t already updated your manifests and configurations to the new registry, check out this quick YouTube video by Justin Garrison, one of our Amazon Web Services Developer Advocates.

Removal of alpha seccomp annotations

The seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io annotations have been removed . Secure computing mode (i.e., “seccomp”) provided improved workload security by restricting system calls for a Pod or individual containers. The alpha seccomp annotations have been deprecated since v1.19, and with their removal in v1.27, seccomp fields will no longer be auto-populated for Pods with seccomp annotations. Instead, you should utilize the securityContext.seccompProfile field for Pods or containers to configure seccomp profiles. To check whether you are using the deprecated alpha seccomp annotations in your cluster, run the following command:

kubectl get pods --all-namespaces -o json | grep -E 'seccomp.security.alpha.kubernetes.io/pod|container.seccomp.security.alpha.kubernetes.io'

Removal of `--container-runtime` command line argument

The --container-runtime command line argument for the kubelet has been removed . The default container runtime for Amazon EKS has been containerd since v1.24 , which eliminates the need to specify the container runtime. It is important that you do not pass this argument to “ --kubelet-extra-args ” in order to prevent errors during the node bootstrap process. You must remove the --container-runtime argument from all your node creation workflows and build scripts. This includes your bootstrap scripts, infrastructure as code (IaC) templates, such as those for eksctl and Terraform , and any custom AMI build scripts, such as those for the Amazon EKS optimized Amazon Linux AMI build script. Let’s consider the following examples for eksctl and Terraform.

eksctl

You must remove the container-runtime from the kubeletExtraConfig field:

nodeGroups:
  - name: your-nodegroup-name
    instanceType: m5.large
    desiredCapacity: 3
    minSize: 1
    maxSize: 4
    kubeletExtraConfig:
      container-runtime: "your-runtime"

Terraform

You must remove the container-runtime from the bootstrap_extra_args field:

  node_groups = {
    eks_nodes = {
      desired_capacity = 2
      max_capacity     = 10
      min_capacity     = 1

      instance_type = "m5.large"
      k8s_labels = {
        Environment = "test"
        Name        = "eks-worker-node"
      }

      additional_userdata = "echo foo bar"
      bootstrap_extra_args = "--container-runtime=your-runtime"
    }

Conclusion

In this post, we walked through the notable changes in Kubernetes version v1.27 and highlighted some of the most exciting features available. Be sure to check out the other improvements documented in Kubernetes v1.27 release notes . If you need assistance with upgrading your cluster to the latest Amazon EKS version, refer to our documentation here .

Amazon EKS provides support for at least four Kubernetes versions at any given time. Given the nature of the Kubernetes release cycle, it’s critical for all customers to have an ongoing upgrade plan. If you’re still running an older version of Kubernetes like 1.22 and 1.23, please consider upgrading to one of the newer supported versions . The end-of-support for 1.22 clusters is June 4, 2023, and the end-of-support for 1.23 clusters will be in October 2023. If you have more questions concerning Amazon EKS version support, then refer to our FAQ .