Discussion: `attempt_number` semantics diverge from Hookdeck API

[alexluong]

Current outpost behavior

• attempt_number is 0-indexed (starts at 0)
• Automated retries increment sequentially (0, 1, 2, ...)
• Manual retries always produce attempt_number=0 because the retry path doesn't have context of how many prior attempts exist for the event+destination pair — so a manual retry after automated retries (0, 1, 2) produces a duplicate attempt_number=0

Hookdeck API behavior

• attempt_number is 1-indexed (starts at 1)
• Both automated and manual retries increment the number

Questions to resolve

1. Indexing

Should we align with Hookdeck's 1-indexed convention?

2. Manual retry incrementing and retry schedule interaction

Should manual retries look up the prior attempt count and continue incrementing? If so, how does this affect the retry schedule? E.g. if the schedule allows 5 retries over 24 hours and a user manually retries 4 times (all failing), does that exhaust the schedule — leaving only 1 automated retry remaining? If manual retries should not count toward the schedule limit, we effectively need two separate concepts: the automated retry counter (used by the scheduler to determine remaining retries) and the total attempt number (including manual).

3. Persistence vs derivation

Currently attempt_number is persisted on the Attempt record. Should we instead derive it at read time (e.g. by ordering attempts for an event+destination by time and assigning a sequence number)? Deriving avoids needing writers to agree on the correct value, but adds read-time complexity.

4. RetryTask.attempt_number reliability

Currently the retry message carries the next attempt_number (set by RetryTaskFromDeliveryTask as task.Attempt + 1). Should the retrymq handler instead calculate it at execution time (e.g. by counting existing attempts)? The current approach is unreliable because manual retries bypass RetryTask entirely, and there's a race condition where a manual and automated retry could both be enqueued to deliverymq — making the carried value stale. This is an extreme edge case but means the value on RetryTask is effectively unreliable.

5. Delivery queue exclusivity (related)

Should we enforce that only one delivery task for the same event+destination can be in the queue at a time? E.g. if an automated retry is pending in deliverymq, reject a manual retry (and vice versa — if a manual delivery task is queued, block automated retries from being enqueued). This would prevent duplicate concurrent deliveries and simplify the attempt_number correctness problem. Without this, we may need to calculate attempt_number during the deliverymq handler by querying the logstore for prior attempts — adding a dependency from deliverymq to logstore.

[alexbouchardd]

Should we align with Hookdeck's 1-indexed convention?

Definitely, not to align the behaviour but because of what it describes. It's the "first" attempt

2. Manual retry incrementing and retry schedule interaction

I would tend to think it shouldn't impact the retry schedule but I think either way is "fine"

3. Persistence vs derivation

When scheduling the retry, could we know what the latest retry was and propagate the retry count as part of the RetryTask and use that value for the log?

The current approach is unreliable because manual retries bypass RetryTask entirely

That seems like the problem really, manual retry should be a RetryTask?

Should we enforce that only one delivery task for the same event+destination can be in the queue at a time?

Ideally, if reasonably doable, yes

[alexbouchardd]

The attempt_number becomes part of the delivery idempotency key really, so for a given attempt_number only a single retry can happen. That's an effective way of deduplicating rather then trying to figure out if an event is already on-going a retry

[alexluong]

Design

Manual retries affect the retry schedule. attempt_number is a single 1-indexed counter across all deliveries (auto + manual).

On failure (auto or manual):

• Cancel any pending scheduled retry
• If budget remains: schedule next auto retry at the backoff tier corresponding to the current attempt_number
• If budget exhausted: cancel any pending scheduled retry, done

On success (auto or manual):

• Cancel any pending scheduled retry, done

Example

Schedule: [30s, 60s, 300s] — max 4 attempts (1 initial + 3 retries)

Attempt 1   auto    failed  → schedule retry: 30s
Attempt 2   manual  failed  → cancel 30s, schedule retry: 60s
Attempt 3   auto    failed  → schedule retry: 300s
Attempt 4   manual  failed  → cancel 300s, exhausted, done
Attempt 5   manual  failed  → cancel if anything pending, exhausted, done

Key decisions

1. Single counter, not two tracks. Users want a simple chronological timeline ("what happened with this delivery"). Auto vs manual is metadata on the attempt, not a separate numbering scheme.

2. Manual retries consume retry budget. A manual retry advances the schedule the same way an auto retry would. This keeps the model simple (one counter drives both the UI and the scheduler).

3. Derive attempt_number from logstore. Manual retries query the logstore for the count of existing attempts at API time, rather than carrying a potentially stale value in the message. This is reliable because manual retries happen on human timescales — well after async log writes complete.

4. Cancel-then-schedule on failure. A failed manual retry replaces the current pending auto retry with the next tier, rather than running alongside it. This prevents duplicate deliveries and keeps the schedule progression predictable.