AWS ASG warm pool support

[mariadb-JeffBachtel]

Which component are you using?: cluster-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.: Scaling up AWS EC2 instances can take some time, with ASG operations

Describe the solution you'd like.: I'd like cluster-autoscaler to be aware of AWS warm pools, a new feature briefly described in https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-ec2-auto-scaling-introduces-warm-pools-accelerate-scale-out-while-saving-money/

Describe any alternative solutions you've considered.:

Additional context.:

[Noksa]

Hello.
Any updates?

[prithulagta]

Hi team, it will be really handy to have this feature as it will reduce the time for Pending pods to get scheduled via cluster autoscaler.

[bitva77]

I agree this would be a great feature to have though thinking it through it may be difficult to implement at this time. Spit balling:

• How would CA identify nodes that are in the Warm Group? At the moment they're not tagged with anything identifiable other than the normal tags defined by the Launch Template out of the ASG. So would CA just assume anything in a Stopped state that's in the ASG is a "Warm Node" that can be spun up again?

• You need graceful node shutdown enabled as well as we the way it currently works, the instance shuts down before K8s can fully terminate pods on the warm node not needed anymore and so pods get stuck.

• Do Warm Nodes ever get terminated by CA? When/why?

I'm sure there's more....

[vijaykumarmcp]

Ive been facing the same issue and have a workaround for the same which goes like this:

1. AWS have instance Lifecycle for all insatnces which starts with "Warmed" for all warmpool insatnces.
2. I am using user-data script which checks whether the instance is a normal or warmed, thanks to insatnce lifecycle.
3. Depending on the check it decide whether to let the node join the cluster or not in the first boot of instance.
4. When the node is called from warmpool to join ASG the Lifecycle changes to Pending and the user-data script is again called at start and then it can let the node join the cluster.

However this workaround has some shortcomings , the nodes might take longer than expected to scale down and the warming of instance might take longer as well.

If the Lifecycle of insatnces can be used to accomodate warmpool feature in cluster autoscaler would be really great.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ryan4yin]

This is a very useful feature.

[ryan4yin]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[jeromeinsf]

/remove-lifecycle rotten

[rshad]

Hi @vijaykumarmcp,

I am not sure if you're facing the same issue described in this issue, but when you say in #4005 (comment):

When the node is called from warmpool to join ASG the Lifecycle changes to Pending and the user-data script is again called at start and then it can let the node join the cluster.

Actually, the warm pool node is never called by the Cluster Autoscaler to join the ASG and this is the problem. So the instance is never started when needed by the Cluster Autoscaler and so the user data script is never triggered.

As I verified, the only way to let the warm pool nodes join the ASG is to modify the desired capacity of the ASG manually. Maybe I'm missing the point in your steps, if so, please let me know.

Thanks!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jeromeinsf]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[anthonyhaussman]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[benmccown]

/remove-lifecycle rotten

[jebbens]

This doesn't make CA fundamentally aware of warm pools, but you can configure a self-managed node group to work with warm pools per example here. Only issue is that because CA terminates specific instances during scale-in using the TerminateInstanceInAutoScalingGroup API, the instance reuse policy is ignored, i.e. does not work.

[robbo10]

This doesn't make CA fundamentally aware of warm pools, but you can configure a self-managed node group to work with warm pools per example here. Only issue is that because CA terminates specific instances during scale-in using the TerminateInstanceInAutoScalingGroup API, the instance reuse policy is ignored, i.e. does not work.

Hi @jebbens - We have been trying to use warm pools with CA, however the node join the cluster in a NotReady status, and CA fails as it doesn't know about the instances.

I was looking at the repo which you linked, we have our own custom launch template EKS module therefore using the above module would not work for us. However from looking through the code is the main takeaway in the userdata script, whereby we would need to update this so that if nodes are part of the warm pool, they don't automatically join the cluster. https://github.com/aws-samples/eks-node-group-with-warm-pool/blob/main/user_data/node-config.tftpl#L111

Should that solve our issue?

Thanks

[jebbens]

Hello @robbo10. Yes, the user-data is key, plus a few other configs. From the first paragraph of the README: "The key components that enable this are the user data, initial lifecycle policy, warm pool configuration, and the additional IAM permissions." Hope that helps!

[cclo-astri]

Hello all,

Is there any update about this feature ?

Thanks.

[cclo-astri]

Hello all,

I just found this which could make use of AWS ASG warm pool with CA, but unfortunately the EC2 instance will still be terminated instead of back to warm pool.

And according to information from here:

Warm pool instance reuse policies do not currently work with CA, which terminates specific nodes/instances and decrements desired capacity of the ASG via the TerminateInstanceInAutoScalingGroup API.

Seems the EC2 termination is invoked by CA internally, but at least the node (EC2) start time is significantly reduced.

Thanks all.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[benmccown]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[benmccown]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[agoddijn-fern]

New to the party but agreed this would be a very useful feature!

/remove-lifecycle stale

[agoddijn-fern]

I came across this on my travels. Used this with a daemonset to pre pull large images and that works pretty well to create "warm" nodes with images pre-pulled (though obviously nodes aren't hibernated or stopped)

Maybe this will help with someone else's use case

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-configure-overprovisioning-with-cluster-autoscaler