Add implementing-crash-safe-work-queues skill (application-development)

[Dhruvjain35]

What does this PR do?

Adds implementing-crash-safe-work-queues to application development: the schema and protocol decisions that let a work queue on CockroachDB survive workers being SIGKILLed mid-operation without duplicating external side effects — and without becoming the queueing hotspot the docs warn about. It treats those as the same problem, because the usual fix for one makes the other worse.

Twelve steps, each a decision with its reasoning:

• Claiming work — one statement: a candidate CTE plus a compare-and-swap on a per-row lease_epoch, so two workers that select the same row cannot both win. SELECT ... FOR UPDATE SKIP LOCKED is treated as an optimization on top of the CAS, not a replacement for it — it is advisory within a transaction and says nothing about a worker that is already gone.
• Keeping the queue off the hot path — a partial claim index on non-terminal states, so completed rows leave the index and it tracks backlog depth rather than table size; state transitions instead of DELETE, so tombstones never accumulate in front of the next scan; and an explicit computed shard column prefixing the index rather than USING HASH, so workers can be assigned shard subsets by a stable application-facing name.
• Lease expiry — the lease is liveness, the fencing token is correctness; a paused worker is not a stopped worker. Folding lease expiry into the same available_at column the claim predicate already reads also removes the reaper process, which is itself a common second hotspot.
• Idempotency keys as GENERATED STORED columns — derived only from immutable columns, so a worker recovering from a crash computes the same key the dead one used and there is no code path that can supply a different one.
• A recovery table with a defined answer per crash window, and an explicit statement that this is effectively-once, not exactly-once.

references/validation-queries.md carries one assertion per property — plan shape, index size against backlog rather than table size, the CAS rejecting a second claimer, the fence rejecting a zombie write, reconciliation against the external ledger. Every property here degrades silently: the rows still come back correct, just from the wrong plan.

Related Issue

No proposal issue. The skill was already written when I read CONTRIBUTING's proposal step, and opening one retroactively seemed like more noise than a PR you can read. Happy to file one if you would rather have the process in order.

Type of Change

• New skill
• Skill update/improvement
• Documentation update
• Infrastructure/tooling change
• Bug fix

Skill Compliance Checklist

• SKILL.md includes required frontmatter fields (name, description)
• Skill name follows naming conventions (lowercase, hyphens, gerund form preferred)
• Skill name matches directory name exactly
• Description is specific and includes "when to use" trigger keywords
• Description is max 1024 characters (850)
• Skill is under 500 lines (390) and uses references/ for detailed content
• References official CockroachDB documentation (not duplicate content)
• Includes safety guardrails for risky operations (if applicable)
• Tested with at least one AI agent (manual validation) — see Testing below
• No time-sensitive information (version numbers, dates, "currently")
• Directory structure follows specification (only scripts/, references/, assets/ subdirs)
• No reserved words in skill name ("anthropic", "claude")
• Local validation passes: python scripts/validate-spec.py skills/

Documentation Updates

• README.md — not needed; it lists domains, not individual skills
• Skill SKILL.md files

Testing

Manual testing:

• Tested with AI agent — the skill was written from a working implementation rather than validated as a loadable skill, so I have not run a discovery test against a skill-loading agent. Flagging it rather than checking the box.
• Validated technical accuracy against CockroachDB docs
• Tested the SQL and the validation queries — the pattern is implemented and crash-tested in the reference implementation linked below, including the plan assertions in references/validation-queries.md

Automated testing:

• Local validation passes. python scripts/validate-spec.py skills/cockroachdb-application-development/implementing-crash-safe-work-queues/ --strict reports zero errors and zero warnings, including the gerund-form heuristic.
• CI validation will run automatically on this PR

Additional Context

The pattern comes from AXIOM, a hackathon project — an agent execution layer on CockroachDB with a test per crash window and a harness that SIGKILLs workers mid-refund and reconciles against the provider's ledger afterwards. It is not deployed in production anywhere; the skill is written as general guidance and the implementation is cited once, in References, as something to read rather than something to trust.

Reviewer Notes

Overlap with #22. That PR proposes designing-durable-agent-execution in this same domain, and the two share three ideas: intent committed before the effect, deterministic idempotency keys, and epoch fencing. They differ in what they are about. That one is a per-run step ledger for an agent; this one is the queue table itself, so most of it — partial index shape, tombstones at the queue head, shard prefixing, claim contention, STORING for an index-only claim, eliminating the reaper — is material it does not cover. The two also disagree on one point worth resolving either way: it says exactly-once, this one argues that no system calling an API it does not control can promise that, and says effectively-once. Better you hear about the overlap from me than find it. Happy to narrow scope, cross-reference, merge the two, or drop this if you would rather have one skill in this space.

The scope boundary against designing-application-transactions is stated in the skill itself: retry loops, backoff, and connection pooling are that skill's job, and this one links to it rather than restating it.

---

By submitting this PR, I confirm:

• I have read the contributing guidelines
• I have followed the Agent Skills Specification
• I have tested my changes
• I am willing to address review feedback