what is the critierium for chosing node pool out of 2 node pools that are almos equal.

[camaeel]

I created 2 nodePools:

• default (using spots)

  apiVersion: karpenter.sh/v1
  kind: NodePool
  metadata:
    creationTimestamp: "2025-02-10T08:14:20Z"
    name: default
  spec:
    disruption:
      budgets:
      - nodes: 10%
      consolidateAfter: 5m
      consolidationPolicy: WhenEmptyOrUnderutilized
    limits:
      cpu: "30"
      memory: 128Gi
    template:
      metadata:
        labels:
          label: karpenter1
      spec:
        expireAfter: 720h
        nodeClassRef:
          group: karpenter.k8s.aws
          kind: EC2NodeClass
          name: test-userdata
        requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
          - t
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values:
          - "1"
        - key: kubernetes.io/arch
          operator: In
          values:
          - arm64
          - amd64
        - key: karpenter.sh/capacity-type
          operator: In
          values:
          - spot
        - key: capacity-spread
          operator: In
          values:
          - "2"
        startupTaints:
        - effect: NoSchedule
          key: node.cilium.io/agent-not-ready
        taints:
        - effect: NoSchedule
          key: label
          value: karpenter1
    weight: 5

• on-demand

  apiVersion: karpenter.sh/v1
  kind: NodePool
  metadata:
    name: on-demand
  spec:
    disruption:
      budgets:
      - nodes: 10%
      consolidateAfter: 5m
      consolidationPolicy: WhenEmptyOrUnderutilized
    limits:
      cpu: "30"
      memory: 128Gi
    template:
      metadata:
        labels:
          label: karpenter1
      spec:
        expireAfter: 720h
        nodeClassRef:
          group: karpenter.k8s.aws
          kind: EC2NodeClass
          name: test-userdata
        requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
          - t
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values:
          - "1"
        - key: kubernetes.io/arch
          operator: In
          values:
          - arm64
          - amd64
        - key: karpenter.sh/capacity-type
          operator: In
          values:
          - on-demand
        - key: capacity-spread
          operator: In
          values:
          - "1"
        startupTaints:
        - effect: NoSchedule
          key: node.cilium.io/agent-not-ready
        taints:
        - effect: NoSchedule
          key: label
          value: karpenter1
    weight: 5

Test workload:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: karpenter-test
  name: karpenter-test
  namespace: karpenter-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: karpenter-test
  strategy: {}
  template:
    metadata:
      labels:
        app: karpenter-test
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - topologyKey: kubernetes.io/hostname
              namespaces:
                - karpenter-test
              labelSelector:
                matchLabels:
                  app: karpenter-test
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: karpenter-test
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: karpenter-test
#        - maxSkew: 1
#          topologyKey: capacity-spread
#          whenUnsatisfiable: ScheduleAnyway
#          labelSelector:
#            matchLabels:
#              app: karpenter-test
      tolerations:
        - key: label
          value: karpenter1
          effect: NoSchedule
      nodeSelector:
        label: karpenter1
      containers:
        - image: nginx
          name: nginx
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: karpenter-test
  namespace: karpenter-test
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: karpenter-test
  unhealthyPodEvictionPolicy: AlwaysAllow

I explicitly commented out capacity-spread topologySpreadConstraint so it is not taken into account and set both to same weight.
My observation is that every time on-demand pool is taken by karpenter. Questions is what is the criterium for choosing one nodePool over another from all the matching ones?

Karpenter version: 1.2.1
K8s version: v1.30.8

[jmdeal]

Karpenter will choose the compatible NodePool with the highest weight. If there are multiple compatible NodePools with equal weight, there is no formal guarantee which Karpenter will choose. Currently, it selects the first NodePool based on alphabetical order, but this is an implementation detail and could change in the future. I wouldn't rely on this behavior.

[camaeel]

@jmdeal I also expected that it would be alphabetical order, but my observations show that it always prefers "on-demand" over "default". I will try with different names.

[jmdeal]

It's reverse alphabetical, but I still wouldn't rely on it given it's an implementation detail. If you need to prioritize one NodePool over another, it should be assigned a different weight.

[camaeel]

@jmdeal Just wanted to investigate and confirm the current behaviour. Thanx