Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hard-coded server type in Hetzner provider will break on 2024-09-06 #7210

Open
apricote opened this issue Aug 27, 2024 · 4 comments · May be fixed by #7211
Open

Hard-coded server type in Hetzner provider will break on 2024-09-06 #7210

apricote opened this issue Aug 27, 2024 · 4 comments · May be fixed by #7211
Assignees
@apricote
Labels
area/cluster-autoscaler area/provider/hetzner Issues or PRs related to Hetzner provider kind/bug Categorizes issue or PR as related to a bug. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@apricote
Copy link
Member

apricote commented Aug 27, 2024 •
edited
Loading

Which component are you using?:

cluster-autoscaler Hetzner provider

/area provider/hetzner
/area cluster-autoscaler

What version of the component are you using?:

Component version: All current versions

What k8s version are you using (kubectl version)?:

Does not matter

What environment is this in?:

Hetzner Cloud

What did you expect to happen?:

The Hetzner Cloud provider should continue to work after 2024-09-06.

What happened instead?:

The Hetzner Cloud provider will stop working on 2024-09-06.

How to reproduce it (as minimally and precisely as possible):

  • Replace the hardcoded server type (code) with a server type that does not exist (ie. xyz123)

  • Start the provider

  • Observe error messages:

    mixed_nodeinfos_processor.go:160] Unable to build proper template node for draining-node-pool: failed to create resource list for node group draining-node-pool error: failed to get machine type xyz123 info error: server type not found
static_autoscaler.go:387] Failed to get node infos for groups: failed to create resource list for node group draining-node-pool error: failed to get machine type xyz123 info error: server type not found

Anything else we need to know?:

The server type cx11 was deprecated on 2024-06-06. It will be removed from the API on 2024-09-06: https://docs.hetzner.cloud/changelog#2024-06-06-old-server-types-with-shared-intel-vcpus-are-deprecated

The server type is hardcoded for a draining-node-pool, which is not actually used anywhere in the provider. It is only added to the list of known node pools.

Two options:

  • Replace cx11 by the replacement type cx22

    This is minimally invasive, but has the same problem that we are hardcoding a value that might change or be deprecated.

  • Remove draining-node-pool completely from the code

    This feels like the clean choice, as this node pool is not used internally. However, this is a user visible change (node pool will disappear from the status config map), so I am not sure if we can backport this to previous releases.
@apricote apricote added the kind/bug Categorizes issue or PR as related to a bug. label Aug 27, 2024
@k8s-ci-robot k8s-ci-robot added area/provider/hetzner Issues or PRs related to Hetzner provider area/cluster-autoscaler labels Aug 27, 2024
@apricote
Copy link
Member Author

/assign

apricote added a commit to hetznercloud/autoscaler that referenced this issue Aug 27, 2024
@apricote
fix(hetzner): unused draining node pool references deprecated server …
f78cba3 
…type

The `cx11` server type was deprecated on 2024-06-06 and will be removed
from the API on 2024-09-06. Once it is removed, the cluster-autoscaler
provider hetzner will not start anymore with the following error
message:

    Failed to get node infos for groups: failed to create resource list for node group draining-node-pool error: failed to get machine type cx11 info error: server type not found

As the node pool `draining-node-pool` is not being used anywhere, this
commit removes it and the hard coded reference to the deprecated server
type.

Fixes kubernetes#7210
@apricote apricote linked a pull request Aug 27, 2024 that will close this issue
Open
@Shubham82
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Aug 27, 2024
@apricote
Copy link
Member Author

Update from internal conversations:

We plan to implement a workaround for known current users of cluster-autoscaler. This will still cause issues for new users unless a new version of cluster-autoscaler is released.

The workaround will only be available for ~2 weeks after new releases are cut.

We will inform impacted customers about this so they can update before it starts breaking.

@apricote
Copy link
Member Author

Information for customers

The Hetzner provider in current versions of cluster-autoscaler has a bug and relies on the CX11 server type, which we will remove from our ordering options on 6 September 2024.

If you try to use the cluster-autoscaler provider after that date, you will see the following error messages:

mixed_nodeinfos_processor.go:160] Unable to build proper template node for draining-node-pool: failed to create resource list for node group draining-node-pool error: failed to get machine type cx11 info error: server type not found
static_autoscaler.go:387] Failed to get node infos for groups: failed to create resource list for node group draining-node-pool error: failed to get machine type cx11 info error: server type not found

The following versions of cluster-autoscaler are affected:

  • ≤1.28.6 (including 1.27 and older)
  • ≤1.29.4
  • ≤1.30.2
  • ≤1.31.0

We depend on the Kubernetes community and the maintainers of cluster-autoscaler to release new versions. We expect that new official versions are released at the end of September.

To bridge the gap until the Kubernetes community releases the new versions, we published alternative container images of cluster-autoscaler that include a patch for the bug. You can use these in your deployment, but we will remove them one month after new official cluster-autoscaler versions become available. We will not provide any other patch releases on this container image repository. Please switch back to the official images as soon as possible.

  • docker.io/hetznercloud/cluster-autoscaler:v1.28.6-hcloud1 (Build Commit)
  • docker.io/hetznercloud/cluster-autoscaler:v1.29.4-hcloud1 (Build Commit)
  • docker.io/hetznercloud/cluster-autoscaler:v1.30.2-hcloud1 (Build Commit)
  • docker.io/hetznercloud/cluster-autoscaler:v1.31.0-hcloud1 (Build Commit)

Existing Users

To prevent disruptions for existing users of the provider, we will keep the CX11 server type available for these accounts. We will remove that prolonged access to the CX11 server type two weeks after the Kubernetes community releases new versions of cluster-autoscaler.

Links

@emrys90 emrys90 mentioned this issue Aug 30, 2024
Closed
@WebSpider WebSpider mentioned this issue Aug 30, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@apricote apricote

Labels
area/cluster-autoscaler area/provider/hetzner Issues or PRs related to Hetzner provider kind/bug Categorizes issue or PR as related to a bug. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(hetzner): deprecated server type will break on 2024-09-06 hetznercloud/autoscaler
3 participants
@apricote @k8s-ci-robot @Shubham82