Support leader election for HA multi-replica deployment

[fseldow]

What would you like to be added?

Ratify v2 should support leader election via controller-runtime to enable safe multi-replica (HA) deployments.

Background

Currently, the Ratify v2 controller-runtime manager does not enable LeaderElection. This means:

• Single replica works fine
• Multiple replicas will have conflicts — specifically, the cert-controller CertRotator (added via rotator.AddRotator) will attempt certificate rotation from all replicas simultaneously, causing Secret write conflicts

Proposed Changes

1. Enable leader election in the manager:

   • Set LeaderElection: true in ctrl.Options
   • Configure LeaderElectionID (e.g., "ratify-leader-election")
   • Configure LeaderElectionNamespace from the pod namespace

2. Integrate with the health probe system (PR feat(health): add /healthz and /readyz endpoints #2559):

   • Register a readiness checker for leader election status using mgr.Elected() channel
   • Non-leader replicas should still report ready (they serve as followers and can handle read traffic)
   • Register a liveness checker that monitors lease renewal health (detect stuck leader that cannot renew lease)

3. Helm chart updates:

   • Add provider.leaderElection.enabled (default: true when replicaCount > 1)
   • Add provider.leaderElection.id for custom lease name
   • Support provider.replicaCount to enable horizontal scaling

Why this matters

• Cert rotation safety: cert-controller CertRotator relies on leader election to ensure only one replica writes Secrets
• HA readiness: Production deployments need multiple replicas for availability
• Graceful failover: Kubernetes can detect leader loss via health probes and restart the pod

Related

• Issue feat: unified health probe port monitoring HTTP server, controller-runtime, and leader election #2562 — Unified health probe (provides the probe infrastructure)
• PR feat(health): add /healthz and /readyz endpoints #2559 — Health probe implementation (provides the checker registry)
• Gatekeeper uses the same pattern: LeaderElection: false for single replica, configurable for multi

Anything else you would like to add?

The health probe infrastructure in PR #2559 already supports registering arbitrary checkers. The leader election checker would follow the same ReadySignal pattern used for the manager readiness check. Implementation example:

// After manager starts, register leader election checker
registry.RegisterReadiness(healthprobe.MustNewChecker("leader-election", func() error {
    select {
    case <-mgr.Elected():
        return nil // this replica is the leader
    default:
        return nil // non-leaders are still "ready" for read traffic
    }
}))

// Liveness: detect stuck lease renewal
registry.RegisterLiveness(healthprobe.MustNewChecker("leader-lease", func() error {
    // monitor lease renewal timestamp vs timeout
    // return error if lease is about to expire without renewal
}))

Reference: Gatekeeper's approach — uses LeaderElection: false but could be enabled; cert-controller's CertRotator respects manager leader election automatically.