Monitoring version compliance of Amazon Elastic Kubernetes Service by using Amazon Web Services Config

Amazon Elastic Kubernetes Services ( Amazon EKS ) provides a managed Kubernetes service, simplifying cluster operations by offloading undifferentiated heavy lifting to Amazon Web Services. With the Kubernetes release cycle of a new release every 4 months, customers have difficulty in keeping their EKS clusters up-to-date, especially across multiple Amazon Web Services accounts. Additionally, keeping track of EKS version will aid your upgrade strategy and allow teams to benefit from new features, design updates, and bug fixes.

A Kubernetes cluster has a control plane and a data plane. The Kubernetes software running in the different planes need updates to remain up-to-date. The control plane updates are managed by Amazon Web Services, so you only have to launch the process for new minor version. For the data plane, Amazon EKS makes the upgrade for the infrastructure layer a straightforward process by providing Amazon EKS optimized Amazon Machine Images (AMI) and the usage of Amazon EKS managed node group (MNG). However, there is a sequence of actions that need to be followed, as described in our best practices guide .

Using Amazon Web Services Config to monitor Amazon EKS

In this post, we will use Amazon Web Services Config , a service that allows you to continually evaluate your resources through rules. First, we will use Amazon Web Services Config to monitor the version of an Amazon EKS control plane with an Amazon Web Services Config managed rule. Next, we will deploy an Amazon Web Services Config Custom Lambda Rule to monitor the version of Amazon Machine Images (AMI) used in your Amazon EKS managed node groups (MNG) for available update. This will provide your operations team the visibility needed to control the risk associated with upgrades and keep your infrastructure up-to-date by monitoring effective versions. To gain visibility across multiple Amazon Web Services accounts, Amazon Web Services Config can be configured to give you a single view using organization-wide aggregator .

Amazon Web Services Config publishes the event to Amazon EventBridge . Using an event pattern , it’s possible to filter only the events of interests. You can then define one of the many available targets in EventBridge to receive those events.

Here’s the architecture diagram of the Amazon Web Services Config rule that we will deploy:

The different components in this solution are :

1. An Amazon Web Services Config managed rule that checks the EKS Control plane version.
2. An Amazon Web Services Config Custom Lambda Rule that monitors new AMI version for our Managed Node Group.
3. All events generated by Amazon Web Services Config are sent to Amazon EventBridge. Through event pattern, you can trigger notifications to stakeholders (email, http, etc.) or forward them to other targets.

Prerequisites:

You will need the following to complete the steps for the Amazon Web Services managed rule section:

• Amazon Web Services Config enabled in your account, if required you can use the 1-click setup directly from the console.

For the Amazon Web Services Config Custom Lambda Rule section, you will need to:

• Install the Amazon Web Services Command Line Interface (Amazon Web Services CLI) .
• Install the Amazon Web Services Serverless Application Model Command Line (Amazon Web Services SAM CLI).
• Install Git .
• Install Node.js and npm .

Monitor EKS control plane version through Amazon Web Services Config managed rules

For Amazon EKS, we can ensure that the version of our control plane is within a supported version by using the eks-cluster-supported-version provided by Amazon Web Services Config. The full list of Amazon Web Services Config managed rules is available in the documentation.

To enable the eks-cluster-supported-version from Amazon Web Services Config:

1. Open the Amazon Web Services Config console.
2. In the navigation pane, choose Rules.
3. Choose Add rule.
4. Under Amazon Web Services Managed Rules , search for EKS.
5. Select eks-cluster-supported-version , choose Next.
6. Under Parameter , in the Value column, input the latest version of EKS you want to consider compliant. At the time of this writing, I’m using 1.26.
7. Choose Next , review your rule and click Add rule.
8. Select the new rule from the list and under Actions , choose Re-evaluate .

This screenshot shows the rule details with three clusters in your account.

Extending Amazon Web Services Config to monitor EKS Managed Node Groups (MNG) by leveraging Amazon Web Services Lambda

The following steps create an Amazon Web Services Config Custom Lambda Rule and enable it for our managed node groups using the code from our GitHub repo .

Installation procedure:

Let’s start by creating the required Lambda using sam-cli tool:

# You can download the zip from github instead of git clone
git clone https://github.com/aws-samples/aws-config-eks-mng-checkupdate.git
cd aws-config-eks-mng-checkupdate
sam build
sam deploy --guided

We suggest config-custom-rule-eks-mng as the value for the Stack Name . Use default for the other options.

Here’s the sample output of this command:

        Deploying with following values
        ===============================
        Stack name                   : config-custom-rule-eks-mng
        Region                       : ca-central-1
        Confirm changeset            : True
        Disable rollback             : False
        Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-xxxxx
        Capabilities                 : ["CAPABILITY_IAM"]
        Parameter overrides          : {}
        Signing Profiles             : {}
…
Successfully created/updated stack - config-eks-ami in ca-central-1

This will create a Lambda as well as the custom rule EKS-MNG-CheckUpdate in Amazon Web Services Config with the resource policy to invoke that Lambda. The IAM role created for Lambda has read-only permission on EKS as well as GetParameter from Amazon Web Services Systems Manager (SSM) for the available version of AMIs.

Periodic trigger type

Amazon Web Services Config rules can be triggered on a periodic schedule or a configuration change. Amazon Web Services Config currently does not provide support for managed node group (MNG). In order to evaluate a MNG resource, we defined an Amazon Web Services Config Custom Lambda Rule that runs periodically. There are multiple periodic choices, from every hour to every 24 hours. Considering this is linked to AMI update frequencies, we chose the 24 hours period through our template. Note that your Amazon Web Services Config costs will increase with more rule evaluations. To learn more, see Amazon Web Services Config pricing page . Choosing the right frequency for your organization’s needs will help you gain the right level of visibility without accumulating unnecessary charges. If rules need to be re-evaluated faster than the periodic check, it’s possible to trigger an on-demand evaluation from the Amazon Web Services Config console. To trigger an on-demand evaluation, select the appropriate rule, and from the action menu, click the Re-evaluate option.

Here’s a view of results provided by the custom rule from the Amazon Web Services Config console:

The current version of the Amazon Web Services Config rule support node running the Amazon EKS-Optimized AMI of Amazon Linux 2 , and Bottlerocket . See the GitHub project for more details on the supported version.

Event pattern to filter events in Amazon EventBridge

If you want to react to any compliance changes happening with our new rules inside EventBridge. Here’s an example of an event pattern that cover rules used today:

{
  "source": ["aws.config"],
  "detail-type": ["Config Rules Compliance Change"],
  "detail": {
    "configRuleName": ["EKS-MNG-CheckUpdate", "eks-cluster-supported-version"]
  }
}

For the targets, it’s possible to select from a wide-range of Amazon Web Services Services and even external API, see the full list for details .

Cleanup

To prevent additional charges to your account, the following steps will delete the resources that were created:

1 – Through the Amazon Web Services Config console. Select the rule eks-cluster-supported-version and from the action menu, use the Delete rule option.

2 – Delete the custom rule and associated resources via the following command:

sam delete --stack-name config-custom-rule-eks-mng

3 – In the EventBridge console, delete the rule that was added (if any).

Conclusion

In this blog post, we have demonstrated how you can track your EKS Version both at the control and data plane level. By leveraging Amazon Web Services Config rules (managed and custom), you can automate the effort needed to track effective version.

To get a centralized view of your clusters and managed node group version, you can set up an organization-wide aggregator to take advantage of Amazon Web Services Config aggregation capabilities . This will help you in keeping up with the Kubernetes release cycle . In planning your upgrade, you will want to consult the EKS version page , which includes information specifics for each version.