Skip to content

Asset upload complete: transient D1 errors answer 409, and retrying a long verification is undocumented #581

Description

@xxchan

What happens

Publishing a cli-binary release (asset ingest protocol v1) from GitHub Actions failed twice in a row on 2026-09-30, at POST /api/apps/{appId}/builds/{buildId}/assets/{assetId}/upload/complete:

  1. cli-v0.1.4. HTTP 409 {"error":"D1_ERROR: Network connection lost.","code":"ASSET_UPLOAD_SEAL_INTENT_FAILED"}. This was a transient D1 failure while inserting the seal intent. The handler resets the attempt to pending, so a retry can succeed, but it answers 409 like a real conflict. Clients can't tell "retry me" from "you did something wrong".
  2. cli-v0.1.5. upload/complete streams the whole object (135 MB, ferry-linux-arm64) through SHA-256 within one request, and it outlasted the client's 180 s timeout. On retry:
    • Re-declaring the same asset (same idempotency key) returned state ≠ ready with upload: null. That shape isn't in the OpenAPI spec.
    • Completing again returns 409 ASSET_UPLOAD_BUSY while the first verification still holds its lease. That isn't documented either, so our client gave up.

Both builds (0.1.4 = version code 1004, 0.1.5 = 1005 on ferry-cli) were left pending; we have since marked them failed.

Suggestions

  • Transient errors get a retryable status. Answer 503, with Retry-After, when the failure is transient (D1 or network errors in seal intent), not 409. Or document which codes are safe to retry.
  • Document the retry contract for asset ingest.
    • What POST …/assets/uploads returns for an asset being verified (state, upload: null).
    • That ASSET_UPLOAD_BUSY means "verification in progress, poll".
    • How to learn when it finishes. Is it re-declaring until state: ready?
  • Consider an asynchronous complete. Accept it with 202 and let the client poll the asset, so large objects don't depend on one long request. At minimum, state the expected verification time per size.

What Ferry does meanwhile

tools/release/src/publish-hands.ts in botiverse/ferry:

  • allows 10 minutes for upload/complete;
  • retries a whole asset upload on 5xx, D1_ERROR and ASSET_UPLOAD_BUSY;
  • when a retry finds the bytes already uploaded (upload: null), only completes again;
  • marks the build failed if it gives up.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions