Instance type diversification
Amazon EC2 Spot Instances offer spare compute capacity available in the AWS Cloud at steep discounts compared to On-Demand prices. EC2 can interrupt Spot Instances with two minutes of notification when EC2 needs the capacity back. You can use Spot Instances for various fault-tolerant and flexible applications. Some examples are analytics, containerized workloads, high-performance computing (HPC), stateless web servers, rendering, CI/CD, and other test and development workloads.
One of the best practices to successfully adopt Spot Instances is to implement Spot Instance diversification as part of your configuration. Spot Instance diversification helps to procure capacity from multiple Spot Instance pools, both for scaling up and for replacing Spot Instances that may receive a Spot Instance termination notification. A Spot Instance pool is a set of unused EC2 instances with the same Instance type, operating system and Availability Zone (for example, m5.large on Red Hat Enterprise Linux in us-east-1a).
Cluster Autoscaler with Spot Instance Diversification
Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when there are pods that fail to run in the cluster due to insufficient resources (Scale Out) or there are nodes in the cluster that have been underutilized for a period of time (Scale In).
When using Spot Instances with Cluster Autoscaler there are a few things that should be considered. One key consideration is, each Auto Scaling group should be composed of instance types that provide approximately equal capacity. Cluster Autoscaler will attempt to determine the CPU, memory, and GPU resources provided by an Auto Scaling Group based on first override provided in an ASG's Mixed Instances Policy. If any such overrides are found, only the first instance type found will be used. See Using Mixed Instances Policies and Spot Instances for details.
When applying Spot diversification best practices to EKS and K8s clusters while using Cluster Autoscaler to dynamically scale capacity, we must implement diversification in a way that adheres to Cluster Autoscaler expected operational mode.
We can diversify Spot Instance pools using two strategies:
- By creating multiple node groups, each of different sizes. For example, a node group of size 4 vCPUs and 16GB RAM, and another node group of 8 vCPUs and 32GB RAM.
- By Implementing instance diversification within the node groups, by selecting a mix of instance types and families from different Spot Instance pools that meet the same vCPUs and memory criteria.
In this workshop we will assume that our cluster node groups should be provisioned with instance types that have 2 vCPU and 4GiB of memory.
We will use amazon-ec2-instance-selector to help us select the relevant instance types and families with sufficient number of vCPUs and RAM.
There are over 350 different instance types available on EC2 which can make the process of selecting appropriate instance types difficult. To make it easier, amazon-ec2-instance-selector, a CLI tool, helps you select compatible instance types for your application to run on. The command line interface can be passed resource criteria like cpus, memory, network performance, and much more and then return the available, matching instance types.
The CLI tool has been pre-installed in your IDE:
Now that you have ec2-instance-selector installed, you can run ec2-instance-selector --help to understand how you could use it for selecting instances that match your workload requirements. For the purpose of this workshop we need to first get a group of instances that meet our target of 2 vCPUs and 4 GB of RAM.
Run the following command to get the list of instances.
Instance Type VCPUs Mem (GiB) Hypervisor Current Gen Hibernation Support CPU Arch Network Performance
------------- ----- --------- ---------- ----------- ------------------- -------- -------------------
c5.large 2 4 nitro true true x86_64 Up to 10 Gigabit
c5a.large 2 4 nitro true false x86_64 Up to 10 Gigabit
c5ad.large 2 4 nitro true false x86_64 Up to 10 Gigabit
c5d.large 2 4 nitro true true x86_64 Up to 10 Gigabit
c6a.large 2 4 nitro true false x86_64 Up to 12.5 Gigabit
c6i.large 2 4 nitro true true x86_64 Up to 12.5 Gigabit
c6id.large 2 4 nitro true true x86_64 Up to 12.5 Gigabit
c6in.large 2 4 nitro true false x86_64 Up to 25 Gigabit
c7a.large 2 4 nitro true false x86_64 Up to 12.5 Gigabit
c7i.large 2 4 nitro true false x86_64 Up to 12.5 Gigabit
We'll use these instances when we define our node group in the next section.
Internally ec2-instance-selector is making calls to the DescribeInstanceTypes for the specific region and filtering the instances based on the criteria selected in the command line, in our case we filtered for instances that meet the following criteria:
- Instances with no GPUs
- of x86_64 Architecture (no ARM instances like A1 or m6g instances for example)
- Instances that have 2 vCPUs and 4 GB of RAM
- Instances of current generation (4th gen onwards)
- Instances that don’t meet the regular expression
t.*to filter out burstable instance types
Your workload may have other constraints that you should consider when selecting instance types. For example. t2 and t3 instance types are burstable instances and might not be appropriate for CPU bound workloads that require CPU execution determinism. Instances such as m5a are AMD Instances, if your workload is sensitive to numerical differences (i.e: financial risk calculations, industrial simulations) mixing these instance types might not be appropriate.