When a certificate creation or renewal fails, the system retries up to 10 times before marking the operation as permanently failed. However:
- There is no backoff logic in place.
- Retries occur on every manager cycle (every 5 minutes in our production setup).
- This results in all 10 retries being completed in about 50 minutes, which is too aggressive and inefficient (Too low for eventual remote issues).
Proposed Solution
Introduce a progressive backoff strategy for retries.
Backoff Intervals
5m, 5m, 10m, 15m, 30m, 1h, 2h, 4h, 8h, 8h
Behavior
- On failure:
- Increment attempt_count.
- Compute next_retry_at = now + backoff_interval[attempt_count].
- Manager will only retry if now >= next_retry_at.
- After 10 attempts, mark as FAILED.
When a certificate creation or renewal fails, the system retries up to 10 times before marking the operation as permanently failed. However:
Proposed Solution
Introduce a progressive backoff strategy for retries.
Backoff Intervals
Behavior
- Increment attempt_count.
- Compute next_retry_at = now + backoff_interval[attempt_count].