What's core check-list using Manaed Node Groups as Spot Instance?

Introduction

Thank you for clicking through to my arcticle. I've been a DevOps engineer for 2 years in dev-team of 7 engineers.

My name is MINSEOK, LEE, but I use Unchaptered as an alias on the interenet. So, you can call me anythings "MINSEOK, LEE" or "Unchaptered" to ask something.

[Notice]
This post is more of a translation of a blog post by "Masatoshi Hayashi" in aws official docs.

Topic

1. Spot Instance's Warning Point and Usage Type

2. Previous Restrictions of Spot Instance as Custom Node Group in Amazon EKS.

3. Now Restrictions of Spot Instance as Managed Node Group in Amazon EKS.

Prerequisites

1. Unchaptered (Blog) | What's the Amazon EKS?

Spot Instance's Warning Point and Usage Type

When use Amazon EC2 Spot Instance, clients can use EC2 with discounts price until capacity pools.

[Warning Point]

1. The EC2 Spot Instance can be interrupted with a 2 minute notification.

[Usage Type]

1. API Endpoint with stateless.

2. Batch Computing

3. Machine Learning Workload.

4. Big Data ETL using Apache Spark

5. Application to process queue.

6. CI/CD Pipeilne

Previous Restrictions of Spot Instance as Custom Node Group in Amazon EKS

Before December 2020, clients must use Custom Node Group to use Spot Instance.

[Restrictions]

1. Set Configuration

2. Maintainence

3. Deploy toos to process Spot Instance's inturruption.

4. Deploy new AMI.

5. Auto Scaling Group

Now Restrictions of Spot Instance as Managed Node Group in Amazon EKS.

1. Use spot instance for fault-tolerant applications

   What is the fault-tolerant applications?

2. Use more than 2 spot instance in same time for availability.

3. Use all availability zones

   Is this instance family supported in the all availability zones?

   Some kind of instance family isn't supported in all availaibility zones.
   So, you can check "Is this instance family is supported in the all availability zones?"

4. Consider EFS(Elastic File System), not using EBS according to your workloads.

EKS managed node groups with Spot Instances: a look under the hood

We've learned some constraints and recommendations.
However, there are a few additional considerations that aren't in this article.

1. Allocation Strategy : Capacity-Optimized.

   When Node Group is scaled out, new instance will be launched from the most-available capacity pools. This works have two advantages.

   1. Decrease the number of spot interruptions in node groups

   2. Increase the resilience of the application.

2. No additional Configuration Tools : Node Termination Handler

   Managed Node Groups handle Spot Interruption

   The underlying Auto Scaling Group is opted-in to Capacity Rebalancing, which means one of spot instances will be elevated risk of interruption and get ec2 instance rebalancing recommendation.

   Managed Node Groups's Instance Family
   The more instance types configured in Managed Node Group, the higher the probaility that EC2 Auto Scale will launch a spot instance.

3. Automatic Pod Balancing in each Availability Zones

   Draining for Balancing
   If node group needs to rebalance capacity between AZs, it will automatically drain the pods from the instances, are being scaled in.

4. Automatic Tagging

   Auto Scaling Group are automatically tagged
   The Kubernetes CAS(Cluster Auto Scaler) auto discovery functionality.

   • k8s.io/cluster-autoscaler/enabled=true

   • k8s.io/cluster-autoscaler/<cluster-name>

Conlusion

After eksctl, version 0.33, you can start
You can start EC2 Spot Instance, after eksctl (version 0.33).

• No configuration overhead

• No operation overhead

When resources are scarce, such as when traffic spikes, you may need to use Kubenetes Karpenter.

References

• AWS Blog | Amazon EKS now supports provisioning and managing EC2 Spot Instances in managed node groups

• AWS Blog | Amazon EKS が、マネージド型ノードグループでの EC2 スポットインスタンスのプロビジョニングと管理をサポート