[CA] Pod pending with Custom Scheduler request for scaling node

[suker200]

I have written a custom-scheduler in scheduling pod based on node workload.

Repo reference

The "--verify-unschedulable-pods=false" option was removed in #189 , which lead to CA will not scale up with custom-scheduler

IMG reference

Do we have any way to trigger CA (api, etc..)?

[MaciekPytel]

Currently CA doesn't support custom schedulers, scheduler extenders, or any other custom way of scheduling that is not done via scheduler predicates in main kubernetes repository. Setting --verify-unschedulable-pods=false (back when the flag existed) did not make it work for custom schedulers. All it would achieve was to make CA add a completely random node (or a few), with no guarantee whatsoever that it will help pending pod.

On the most basic level CA works by simulating how a scheduler would behave if cluster looked differently. This assumes that the scheduler code baked into CA is the same as the one used by scheduler.

Depending on what exactly your scheduler does you may be able to get it to work with CA by modifying https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/simulator/predicates.go (perhaps by implementing and injecting your own predicate?) and building your own CA image.

[suker200]

Hi, Thank for you comment. I will give it a try.

Do we have any plan for supporting custom-scheduler feature?

[MaciekPytel]

Not really, at least in the nearest future. There are 2 main issues we would have to solve:

1. The scheduler works in real cluster. You don't pass the state of the cluster to it, it just takes it from informer (more generally: apiserver). That's not true for our use-case - we need to know what it would do in a simulated cluster, with some nodes added or removed and some pods shuffled around. And we need to run against different simulations all the time, so it must be stateless.
2. Performance. We need to do much, much more scheduling than scheduler does. We only run scheduler predicates, not priorities and we even have a bunch of custom optimizations for predicates (for example: Figure out and implement custom handling for MatchInterPodAffinity predicate #257). The main issue here goes back to point 1 - how would we pass complete cluster state for all of those predicates we need to run? Serializing and sending to a separate process complete cluster state for each query doesn't seem feasible. Unless we can figure out a way around this, we won't be able to support anything outside of CA binary.
   Perhaps some sort of golang interface that could be implemented by users and compiled with CA could be an option? Either way it's not something we're working on right now.

[suker200]

Thank, I will close this issue and try your suggestion. I will update back if there has news.