Introduction
In November 2021, Amazon Web Services introduced
Karpenter
, an open-source high-performance Kubernetes Cluster Autoscaler licensed under the
Apache License 2.0
. Karpenter helps improve your application availability and cluster efficiency by rapidly launching right-sized compute resources in response to changing application load. Since its release, we’ve been seeing an increase in customers migrating from
Kubernetes Cluster Autoscaler
to Karpenter. However, for customers running a heterogeneous Amazon Elastic Kubernetes Service (
Amazon EKS
) cluster with Windows workloads, it became a showstopper as Karpenter didn’t support Windows nodes, until now.
The OSS
community
did a great job starting development on Windows workloads in Karpenter. The Amazon Web Services team took it a step further to review the proposed design, add enhancements to improve the customer experience, and integrate it with our internal continuous integration (CI) process.
When Karpenter is installed in your cluster, it observes the aggregate resource requests of unscheduled pods and decides to launch new nodes when additional capacity is needed, while deciding to deprovision nodes when that capacity is no longer needed. By doing this, Karpenter reduces the scheduling latencies and infrastructure costs of your cluster.
Figure 1: Karpenter high-level scheduling
In this post, we focus on scaling out/in Windows Server 2019 and Windows Server 2022 using Karpenter for Amazon EKS. To learn more about Karpenter architecture and components, access the Karpenter website.
Prerequisites
- Ensure you are running eksctl commands with an Amazon Web Services Identify and Access Management (
Amazon Web Services IAM)
profile that has permissions to create and manage
Amazon EKS
. This Amazon Web Services IAM security principal is used in the Getting Started section below for the Amazon Web Services Command Line Interface (
Amazon Web Services CLI
) configuration.
- Ensure you are using
eksctl v0.124.0
or higher to operate Karpenter.
- Follow the
Getting Started section in the Amazon EKS documentation
to install
aws cli
,
kubectl
, and
eksctl
on your development machine.
- Alternatively, you could leverage
Cloud9
or
Cloudshell
to handle deployment and maintenance tasks.
Solution overview
- Create OS variables to be used throughout the post.
- Deploy Karpenter service requirements.
- Create an Amazon EKS cluster with the necessary iamIdentityMappings for Karpenter.
- Enable Amazon EKS Windows support.
- Install Karpenter with Helm.
- Create Karpenter provisioner and NodeTemplate.
- Test Karpenter for Windows – scale out.
- Test Karpenter for Windows – scale in.
- Cleanup test resources.
Walkthrough
1. Create OS variables to be used throughout the post
export KARPENTER_VERSION=v0-c990a2d9fb10c1bfeffd5c6af64bf8575536d67e
export AWS_PARTITION="aws"
export CLUSTER_NAME="windows-karpenter-demo"
export AWS_DEFAULT_REGION="us-west-2"
export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export TEMPOUT=$(mktemp)
2. Create Karpenter service requirements
Karpenter directly integrates with the Amazon Elastic Compute Cloud (
Amazon EC2
) API endpoint to take specific actions based on events such as spot interruption or instance state changes. The following command automatically deploys the necessary Amazon Web Services services/components such as
Amazon EventBridge
rules applied to messages being sent over an Amazon SQS queue using
Amazon Web Services CloudFormation
.
curl -fsSL https://karpenter.sh/v0.29/getting-started/getting-started-with-karpenter/cloudformation.yaml > $TEMPOUT \
&& aws cloudformation deploy \
--stack-name "Karpenter-${CLUSTER_NAME}" \
--template-file "${TEMPOUT}" \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides "ClusterName=${CLUSTER_NAME}"
Upon successful execution of the
Amazon Web Services CloudFormation template
, you’ll be presented with the following output:
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - Karpenter-windows-karpenter-demo
3. Create an Amazon EKS cluster with the necessary iamIdentityMappings for Karpenter
Next, we deploy a temporary Amazon EKS cluster using
eksctl
in order to test Karpenter integration with Windows. The necessary Amazon Web Services IAM and IdentityMapping are created as ServiceAccounts and added to the Kubernetes ConfigMap.
eksctl create cluster -f - <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ${AWS_DEFAULT_REGION}
version: "1.27"
tags:
karpenter.sh/discovery: ${CLUSTER_NAME}
iam:
withOIDC: true
serviceAccounts:
- metadata:
name: karpenter
namespace: karpenter
roleName: ${CLUSTER_NAME}-karpenter
attachPolicyARNs:
- arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}
roleOnly: true
iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
managedNodeGroups:
- instanceType: m5.large
amiFamily: AmazonLinux2
name: ${CLUSTER_NAME}-linux-ng
desiredCapacity: 2
minSize: 1
maxSize: 10
EOF
export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output text)"
export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
echo $CLUSTER_ENDPOINT $KARPENTER_IAM_ROLE_ARN
eksctl uses Amazon Web Services CloudFormation to create all the necessary resources to build an Amazon EKS cluster. Upon successful creation of your cluster, you’ll be presented with a similar output. If the cluster creation fails, then the failure reason is provided in the Amazon Web Services CLI output (or Amazon Web Services CloudFormation console).
2023-06-14 06:20:19 [✔] all EKS cluster resources for "windows-karpenter-demo" have been created
2023-06-14 06:20:19 [ℹ] nodegroup "windows-karpenter-demo-linux-ng" has 2 node(s)
2023-06-14 06:20:19 [ℹ] node "ip-192-168-12-160.ec2.internal" is ready
2023-06-14 06:20:19 [ℹ] node "ip-192-168-53-156.ec2.internal" is ready
2023-06-14 06:20:19 [ℹ] waiting for at least 1 node(s) to become ready in "windows-karpenter-demo-linux-ng"
2023-06-14 06:20:19 [ℹ] nodegroup "windows-karpenter-demo-linux-ng" has 2 node(s)
2023-06-14 06:20:19 [ℹ] node "ip-192-168-12-160.ec2.internal" is ready
2023-06-14 06:20:19 [ℹ] node "ip-192-168-53-156.ec2.internal" is ready
2023-06-14 06:20:20 [ℹ] kubectl command should work with "/Users/bpfeiff/.kube/config", try 'kubectl get nodes'
2023-06-14 06:20:20 [✔] EKS cluster "windows-karpenter-demo" in "us-east-1" region is ready
4. Enable Amazon EKS Windows support
To deploy Windows nodes to our cluster, we need to enable
Amazon EKS Windows support
.
kubectl apply -f - <<EOF
---
apiVersion: v1
kind: ConfigMap
metadata:
name: amazon-vpc-cni
namespace: kube-system
data:
enable-windows-ipam: "true"
EOF
5. Install Karpenter with Helm
Next, we will use
Helm
to install Karpenter.
# Logout of helm registry to perform an unauthenticated pull against the public ECR
helm registry logout public.ecr.aws
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \
--set settings.aws.clusterName=${CLUSTER_NAME} \
--set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
--set settings.aws.interruptionQueueName=${CLUSTER_NAME} \
--set controller.resources.requests.cpu=1 \
--set controller.resources.requests.memory=1Gi \
--set controller.resources.limits.cpu=1 \
--set controller.resources.limits.memory=1Gi \
--wait
Upon successful installation, you‘ll see the following output.
Release "karpenter" does not exist. Installing it now.
Pulled: public.ecr.aws/karpenter/karpenter:v0-c990a2d9fb10c1bfeffd5c6af64bf8575536d67e
Digest: sha256:33e2597488e3359653515bb7bd43a4ed6c1e811cb95c261175f8808a9ea4fc97
NAME: karpenter
LAST DEPLOYED: Wed Jun 14 08:16:36 2023
NAMESPACE: karpenter
STATUS: deployed
REVISION: 1
TEST SUITE: None
6. Create provisioner as required
Now we create two Karpenter
provisioners
to support Windows Server 2019 and Windows Server 2022 in the same Amazon EKS cluster. The Karpenter provisioner sets constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes.
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: windows2019
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
- key: kubernetes.io/os
operator: In
values: ["windows"]
limits:
resources:
cpu: 1000
providerRef:
name: windows2019
ttlSecondsAfterEmpty: 30
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: windows2019
spec:
subnetSelector:
karpenter.sh/discovery: ${CLUSTER_NAME}
securityGroupSelector:
karpenter.sh/discovery: ${CLUSTER_NAME}
amiFamily: Windows2019
metadataOptions:
httpEndpoint: enabled
httpProtocolIPv6: disabled
httpPutResponseHopLimit: 2
httpTokens: required
---
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: windows2022
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["on-demand"]
- key: kubernetes.io/os
operator: In
values: ["windows"]
limits:
resources:
cpu: 1000
providerRef:
name: windows2022
ttlSecondsAfterEmpty: 30
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: windows2022
spec:
subnetSelector:
karpenter.sh/discovery: ${CLUSTER_NAME}
securityGroupSelector:
karpenter.sh/discovery: ${CLUSTER_NAME}
amiFamily: Windows2022
metadataOptions:
httpEndpoint: enabled
httpProtocolIPv6: disabled
httpPutResponseHopLimit: 2
httpTokens: required
EOF
7. Scale out the deployment
We now have our Amazon EKS cluster prepped for running Windows nodes and all the necessary components of Karpenter. We scale a sample application to see Karpenter automatically add nodes to the Amazon EKS cluster based on demand.
7.1 Run the following code to create your Windows Server 2022 sample application.
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: windows-server-iis-simple-2022
spec:
selector:
matchLabels:
app: windows-server-iis-simple-2022
tier: backend
track: stable
replicas: 0
template:
metadata:
labels:
app: windows-server-iis-simple-2022
tier: backend
track: stable
spec:
containers:
- name: windows-server-iis-simple-2022
image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022
imagePullPolicy: IfNotPresent
command:
- powershell.exe
- -command
- while(1){sleep 2; ping -t localhost;}
nodeSelector:
kubernetes.io/os: windows
node.kubernetes.io/windows-build: 10.0.20348
EOF
The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows Server versions in the same cluster, then you should set additional node labels and nodeSelector fields. Kubernetes automatically adds a label to the Windows node, named
node.kubernetes.io/windows-build
to simplify this.
This label reflects the Windows major, minor, and build number that need to match for compatibility. Here are values used for each Windows Server version:
|
|
Product Name |
Version |
| 1 |
Windows Server 2019 |
10.0.17763 |
| 2 |
Windows Server 2022 |
10.0.20348 |
Based on the build version specified in the Pod nodeSelector, Karpenter launches new Windows nodes with the operating system accordingly. For example, if the build version is specified as 10.0.17763, then Karpenter uses the Windows 2019 provisioner to launch Windows nodes. For more information, please refer to the
Guide for Running Windows Containers in Kubernetes
.
7.2 Run the following command to scale your Windows Server 2022 sample application.
kubectl scale deployment windows-server-iis-simple-2022 --replicas 10
7.3 You can use the Karpenter logs to track the scaling progress.
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
The following output shows the Windows Server 2022 Karpenter provisioner scaling from 0 nodes to 1 to support the 10 replicas we requested be run.
2023-06-14T12:19:01.581Z INFO controller.machine_lifecycle launched machine
{"commit": "c990a2d", "machine": "windows2022-4hq46", "provisioner": "windows2022",
"provider-id": "aws:///us-east-1f/i-039507775a01898e6", "instance-type": "c6a.xlarge",
"zone": "us-east-1f", "capacity-type": "on-demand", "allocatable": {"cpu":"3920m",
"ephemeral-storage":"44Gi","memory":"6012Mi","pods":"110","vpc.amazonaws.com/
PrivateIPv4Address":"14"}}
7.4 Run the following command to track the deployment progress of your pods.
kubectl rollout status deploy/windows-server-iis-simple-2022
You’ll see the 10 replicas being created on our new Karpenter provisioned Windows worker nodes.
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 0 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 1 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 2 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 3 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 4 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 5 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 6 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 7 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 8 of 10 updated replicas are available...
Waiting for deployment "windows-server-iis-simple-2022" rollout to finish: 9 of 10 updated replicas are available...
deployment "windows-server-iis-simple-2022" successfully rolled out
7.5 Run the following code to scale out your Windows Server 2019 deployment.
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: windows-server-iis-simple-2019
spec:
selector:
matchLabels:
app: windows-server-iis-simple-2019
tier: backend
track: stable
replicas: 0
template:
metadata:
labels:
app: windows-server-iis-simple-2019
tier: backend
track: stable
spec:
containers:
- name: windows-server-iis-simple-2019
image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
imagePullPolicy: IfNotPresent
command:
- powershell.exe
- -command
- while(1){sleep 2; ping -t localhost;}
nodeSelector:
kubernetes.io/os: windows
node.kubernetes.io/windows-build: 10.0.17763
EOF
7.6 Run the following command to scale your Windows Server 2019 sample application.
kubectl scale deployment windows-server-iis-simple-2019 --replicas 10
A new Windows Server 2019 worker is launched by Karpenter as more pods are requested to be scheduled. This process is identical to Windows Server 2022 and you can reuse the steps above to track the progress of launching Windows Server 2019 worker node.
8. Scale in the deployment
Karpenter handles scale out and scale in of Windows nodes based on demand. We’ll now tear down our sample applications and watch Karpenter terminate our Windows nodes.
8.1 Run the following commands to delete your sample application deployments.
kubectl delete deployment windows-server-iis-simple-2022
kubectl delete deployment windows-server-iis-simple-2019
The Windows instances launched earlier by Karpenter will now be terminated. You can use the Karpenter logs to track the scale down progress.
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
Once all pods have been terminated, Karpenter deletes all idle instances.
2023-06-20T16:27:12.878Z DEBUG controller.node added TTL to empty node {"commit": "c990a2d", "node": "ip-192-168-99-4.ec2.internal", "provisioner": "windows2022"}
2023-06-20T16:27:15.140Z DEBUG controller.node added TTL to empty node {"commit": "c990a2d", "node": "ip-192-168-88-252.ec2.internal", "provisioner": "windows2019"}
2023-06-20T16:27:42.051Z INFO controller.deprovisioning deprovisioning via emptiness delete, terminating 1 machines ip-192-168-99-4.ec2.internal/c6a.xlarge/on-demand {"commit": "c990a2d"}
2023-06-20T16:27:42.138Z INFO controller.termination cordoned node {"commit": "c990a2d", "node": "ip-192-168-99-4.ec2.internal"}
2023-06-20T16:27:42.478Z INFO controller.termination deleted node {"commit": "c990a2d", "node": "ip-192-168-99-4.ec2.internal"}
2023-06-20T16:27:42.751Z INFO controller.machine_termination deleted machine {"commit": "c990a2d", "machine": "windows2022-4hq46", "node": "ip-192-168-99-4.ec2.internal", "provisioner": "windows2022", "provider-id": "aws:///us-east-1f/i-039507775a01898e6"}
2023-06-20T16:27:54.105Z INFO controller.deprovisioning deprovisioning via emptiness delete, terminating 1 machines ip-192-168-88-252.ec2.internal/c6a.xlarge/on-demand {"commit": "c990a2d"}
2023-06-20T16:27:54.177Z INFO controller.termination cordoned node {"commit": "c990a2d", "node": "ip-192-168-88-252.ec2.internal"}
2023-06-20T16:27:54.480Z INFO controller.termination deleted node {"commit": "c990a2d", "node": "ip-192-168-88-252.ec2.internal"}
2023-06-20T16:27:54.754Z INFO controller.machine_termination deleted machine {"commit": "c990a2d", "machine": "windows2019-khmc5", "node": "ip-192-168-88-252.ec2.internal", "provisioner": "windows2019", "provider-id": "aws:///us-east-1a/i-0978aeb1680f37d7c"}
2023-06-20T16:31:21.596Z DEBUG controller.awsnodetemplate discovered subnets {"commit": "c990a2d", "awsnodetemplate": "windows2019", "subnets": ["subnet-05d7fed709f082b75 (us-east-1a)", "subnet-0109ebad1a6808805 (us-east-1f)", "subnet-0ff0ebe5e1a8630f1 (us-east-1a)", "subnet-0d01b14a3e9c91d1f (us-east-1f)"]}
2023-06-20T16:33:19.192Z DEBUG controller.deprovisioning discovered subnets {"commit": "c990a2d", "subnets": ["subnet-05d7fed709f082b75 (us-east-1a)", "subnet-0109ebad1a6808805 (us-east-1f)", "subnet-0ff0ebe5e1a8630f1 (us-east-1a)", "subnet-0d01b14a3e9c91d1f (us-east-1f)"]} discovered instance types {"commit": "c990a2d", "count": 649}
Cleaning up
When you’ve finished, clean up the resources associated with the example cluster deployment to avoid incurring unwanted charges.
eksctl delete cluster --name ${CLUSTER_NAME} --region us-west-2
If this command times out, then you can run the command above again to show the cluster has been successfully removed.
Conclusion
In this post, we showed you can leverage Karpenter to seamlessly scale out/in your Windows worker nodes on Amazon EKS. Customers no longer need to maintain two auto-scaler solutions on a heterogeneous Amazon EKS cluster with Windows and Linux nodes.
A big shout-out to
topikachu
, who proactively started the development of the add-on.