Add retrying-agent-decisions-under-contention skill (Application Development)

[rajj28]

What does this PR do?

Adds retrying-agent-decisions-under-contention to Application Development.

It covers what to do when a serialization failure invalidates a decision a model has already made. Under SERIALIZABLE an agent's write hits 40001 like any other, but the standard remedy — retry the transaction — behaves differently when the value being written was chosen by a model call that cannot be replayed inside the retry. The retry re-reads state and then writes the decision computed against the previous read.

The motivation came from this repository. Reading designing-application-transactions first showed that two pieces of its guidance are individually correct and, composed for an agent, produce a silent bug:

• step 14 — keep RPC calls outside the transaction (right: a model call inside one holds it open for seconds)
• step 3 — retry the unit of work (right)

Together, for an agent, they mean the retry re-executes the write with a stale decision. Nothing catches it: the transaction is properly serializable, state was read correctly on every attempt, and the committed value is one the model genuinely produced. Only an instrument on the reasoning step distinguishes "re-decided" from "replayed".

The skill links to designing-application-transactions for transaction scoping, retry/backoff and pooling rather than restating any of it.

Its seven sections:

1. Check first whether the constraint belongs in SQL instead — so this skill is not reached for unnecessarily
2. Keep the model call outside the transaction without thereby replaying a stale decision
3. Re-derive rather than replay on retry
4. Refresh retrieved context as well as rows — including where retrieval must sit relative to BEGIN/ROLLBACK, since retrieval inside the racing transaction joins its refresh span
5. Count reasoning invocations, not just retries — the assertion that makes the failure detectable
6. Bound the action space and validate the model's output — a tool schema's enum is guidance, not a contract
7. Record what the agent believed, not only what it wrote

Evidence behind the guidance

Measured on a live CockroachDB Cloud cluster: 250 runs, 5,000 agent decisions, five arms differing by one flag. Runs breaking a SUM(...) <= limit invariant went 47/50 → 0/50 when each attempt re-derived its decision instead of replaying it, with no other change. Repository, methodology log and the two predictions we falsified: https://github.com/rajj28/racelab

Related Issue

Closes #25

Type of Change

• New skill
• Skill update/improvement
• Documentation update
• Infrastructure/tooling change
• Bug fix

Skill Compliance Checklist

• SKILL.md includes required frontmatter fields (name, description)
• Skill name follows naming conventions (lowercase, hyphens, gerund form preferred)
• Skill name matches directory name exactly
• Description is specific and includes "when to use" trigger keywords
• Description is max 1024 characters
• Skill is under 500 lines (305)
• References official CockroachDB documentation (not duplicate content)
• Includes safety guardrails for risky operations (if applicable)
• Tested with at least one AI agent (manual validation)
• No time-sensitive information (version numbers, dates, "currently")
• Directory structure follows specification (only scripts/, references/, assets/ subdirs)
• No reserved words in skill name ("anthropic", "claude")
• Local validation passes: python scripts/validate-spec.py skills/

One validator warning, and why it is a false positive

WARNING: Consider using gerund form (verb-ing) for skill name:
  'retrying-agent-decisions-under-contention'
  → 'retrying-agent-decisions-under-contentioning'

The name is already gerund — it begins with retrying. The heuristic inspects the trailing word, so its suggestion is ...contentioning. The same warning fires on several already-merged skills (enabling-cmek-encryption, managing-tls-certificates, hardening-user-privileges, …), so this looks like a known rough edge rather than something to fix in the name. Happy to rename if maintainers prefer.

python scripts/validate-spec.py skills/ reports 0 errors across the whole tree with this skill added.

Documentation Updates

• README.md
• CONTRIBUTING.md
• Skill SKILL.md files
• Other (specify):

No index file lists skills individually, so nothing else needed updating. Say the word if I have missed one.

Testing

Manual testing:

• Tested with AI agent (specify which one): Claude (Sonnet 4.5 and Opus), via Claude Code with the skill loaded from disk
• Verified skill is discoverable with appropriate prompts — triggers on "my agent retries and writes a stale value", "40001 in an LLM write path", "the policy the agent read changed mid-transaction"
• Validated technical accuracy against CockroachDB docs
• Tested any scripts or commands included in the skill — the Python patterns are reduced from a working implementation that runs against CockroachDB Cloud in the linked repository

Automated testing:

• Local validation script passes: python scripts/validate-spec.py skills/
• CI validation will run automatically on this PR

Notes for reviewers

Two things I would especially value a maintainer's view on:

1. Domain placement. Application Development felt right — it is about how you write the transaction — but a case could be made for Performance and Scaling, since contention is the trigger.
2. The relationship to designing-application-transactions. I have deliberately kept this skill narrow and linked out rather than duplicating. If you would rather this be a section in that skill than a sibling to it, I am glad to reshape it.