feat: sharded notes refs for reduced push contention#818
Draft
svarlamov wants to merge 9 commits intofix/notes-merge-fallback-and-retryfrom
Draft
feat: sharded notes refs for reduced push contention#818svarlamov wants to merge 9 commits intofix/notes-merge-fallback-and-retryfrom
svarlamov wants to merge 9 commits intofix/notes-merge-fallback-and-retryfrom
Conversation
svarlamov
force-pushed
the
feat/sharded-notes-refs
branch
from
2a4ec88 to
cffeccb
Compare
svarlamov
changed the base branch from
main
to
fix/notes-merge-fallback-and-retry
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+357
to
+360
| match exec_git(&shard_fetch) { | ||
| Ok(_) => { | ||
| shards_found = true; | ||
| } |
Contributor
There was a problem hiding this comment.
🟡 False positive NotesExistence::Found when shard wildcard fetch matches nothing
fetch_authorship_notes sets shards_found = true whenever the shard wildcard git fetch exits successfully (line 359). However, as the code's own comment on line 362 notes, "wildcard refspecs that match nothing don't error" — git fetch origin +refs/notes/ai-s/*:…* returns exit code 0 even when no shard refs exist on the remote. This means when sharded_notes is enabled and the remote has no notes at all (neither legacy nor shard), shards_found is true, the !legacy_found && !shards_found guard at line 368 is bypassed, and the function returns NotesExistence::Found instead of the correct NotesExistence::NotFound. This surfaces as an incorrect "notes_existence": "found" in the JSON response of the fetch-authorship-notes command.
Prompt for agents
In src/git/sync_authorship.rs, the `fetch_authorship_notes` function around lines 357-365, `shards_found` is set to `true` on any successful shard fetch, but a wildcard fetch that matches no remote refs also succeeds (exit 0). To fix this, after the shard fetch succeeds, verify that at least one shard tracking ref was actually created. For example, call `resolve_shard_ref_pairs(repository, remote_name)` (which is already defined in this file) and check if the result is non-empty before setting `shards_found = true`. Alternatively, check if any refs matching the `shard_tracking_ref_prefix(remote_name)` pattern exist locally via a quick `for-each-ref` call.
Was this helpful? React with 👍 or 👎 to provide feedback.
Introduces an opt-in sharded notes architecture that splits refs/notes/ai into 256 independent shard refs (refs/notes/ai-s/XX), keyed by the first two hex chars of the annotated commit SHA. This dramatically reduces push contention on large monorepos by narrowing the conflict window from all notes to ~1/256th. Design: - Dual-write: always writes to legacy refs/notes/ai AND shard ref - Union-read: checks shard ref first, falls back to legacy - Sync: fetches/pushes both legacy and shard wildcard refspecs - Backward-compatible: old clients read/write legacy ref as before - Gated behind `sharded_notes` feature flag (default off) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-shard git-notes-merge loop with a batched approach: - 2 git processes to discover+resolve all shard pairs (for-each-ref + cat-file --batch-check) - Skip unchanged shards (tracking OID == local OID) at zero cost - Batch all "copy" operations into one update-ref --stdin call - Only run git notes merge for genuinely diverged shards (typically 1-2 per fetch) Previously: up to 3N git processes (N = number of shards: ref_exists × 2 + merge) Now: 2 + 1 (batch copy) + M (only diverged merges, typically 1-2) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Returns fallback "00" shard instead of panicking if SHA is < 2 chars. Includes debug_assert for development visibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When sharding is enabled, fetch legacy and shard refs as separate git fetch calls. This prevents a missing legacy ref on the remote from blocking shard fetching (and vice versa), which matters for forward compatibility if we ever deprecate the legacy ref. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
svarlamov
force-pushed
the
feat/sharded-notes-refs
branch
from
cffeccb to
3d50a91
Compare
svarlamov
changed the base branch from
fix/notes-merge-fallback-and-retry
to
main
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
svarlamov
force-pushed
the
feat/sharded-notes-refs
branch
from
5c5f6c8 to
9faf183
Compare
svarlamov
changed the base branch from
main
to
fix/notes-merge-fallback-and-retry
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
refs/notes/aiinto 256 independent shard refs (refs/notes/ai-s/XX), keyed by first 2 hex chars of annotated commit SHAsharded_notesfeature flag (default off)Test plan
test_sharded_notes_dual_write_on_push— verifies both legacy and shard refs reach upstream with identical contenttest_sharded_notes_union_read_legacy_fallback— verifies blame works via legacy fallback when no shard refs existtest_sharded_notes_mixed_team_interop— verifies non-sharded clone can read notes pushed by sharded clone🤖 Generated with Claude Code