Automatic Node Provisioning

This workshop has been deprecated and archived. The new Amazon EKS Workshop is now available at www.eksworkshop.com.

With Karpenter now active, we can begin to explore how Karpenter provisions nodes. In this section we are going to create some pods using a deployment we will watch Karpenter provision nodes in response.

In this part off the workshop we will use a Deployments with the pause image. If you are not familiar with Pause Pods you can read more about them here.

Run the following command and try to answer the questions below:

cat <<EOF > inflate.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      nodeSelector:
        intent: apps
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          resources:
            requests:
              cpu: 1
              memory: 1.5Gi
EOF
kubectl apply -f inflate.yaml

Challenge

You can use Kube-ops-view or just plain kubectl cli to visualize the changes and answer the questions below. In the answers we will provide the CLI commands that will help you check the resposnes. Remember: to get the url of kube-ops-view you can run the following command kubectl get svc kube-ops-view | tail -n 1 | awk '{ print "Kube-ops-view URL = http://"$4 }'

Answer the following questions. You can expand each question to get a detailed answer and validate your understanding.

1) Why did Karpenter not scale the cluster after making the initial deployment ?

Click here to show the answer

2) How would you scale the deployment to 1 replicas?

Click here to show the answer

3) Which instance type did Karpenter use when increasing the instances ? Why that instance ?

Click here to show the answer

4) What are the new instance properties and Labels ?

Click here to show the answer

5) Why did the newly created inflate pod was not scheduled into the managed node group ?

Click here to show the answer

6) How would you scale the number of replicas to 10? What do you expect to happen? Which instance types were selected in this case ?

Click here to show the answer

8) How would you scale the number of replicas to 0? what do you expect to happen?

Show me the answers

What Have we learned in this section :

In this section we have learned:

  • Karpenter scales up nodes in a group-less approach. Karpenter select which nodes to scale , based on the number of pending pods and the Provisioner configuration. It selects how the best instances for the workload should look like, and then provisions those instances. This is unlike what Cluster Autoscaler does. In the case of Cluster Autoscaler, first all existing node group are evaluated and to find which one is the best placed to scale, given the Pod constraints.

  • Karpenter uses cordon and drain best practices to terminate nodes. The configuration of when a node is terminated can be controlled with ttlSecondsAfterEmpty

  • Karpenter can scale-out from zero when applications have available working pods and scale-in to zero when there are no running jobs or pods.

  • Provisioners can be setup to define governance and rules that define how nodes will be provisioned within a cluster partition. We can setup requirements such as karpenter.sh/capacity-type to allow on-demand and spot instances or use karpenter.k8s.aws/instance-size to filter smaller sizes. The full list of supported labels is available here