[kriszyp]

⊙ Problem

Operations and replication workers cannot rely on application residency hooks to enforce record ownership. Owners also need complete recovery sources before resuming sharded writes.

🤔 Scope and recovery contract

The bounded contract is intentional: owned snapshots require a certified co-owner or complete backup. A node that loses its origin audit must use a fresh root and new name for snapshot recovery. Same-name recovery requires its complete original audit. A missing frozen start holds the whole database stream, including ordinary tables; stop all database writes and preserve unique local data before replacing the root.

💡 Solution

Support primary-key hashing only; refuse by and slotFrom with #1178. Use the persisted @sharded policy and database map on every thread. Negotiate placement and recovery capability, certify initial owner baselines, and expose recovery holds in cluster_status. Keep retired-origin logs relayable after membership removal.

All three PRs remain draft; the closing scope is prepared for Kris to promote after CI and review.

Refs #980 and #436.

Depends-on: harper#3527

Companions: core #3527, docs #753.

⚖️ Alternatives

Public gate and retained experimental coverage

Supported scope: primary-key hashing only. Schema load refuses by and slotFrom with P1/v5.4 concurrent-move design #1178. The retained implementation is behind the undocumented test-only replication.experimentalAttributeSlots switch; actual attribute placement additionally requires native usesAbortMarkers and enabled storage.rocks.transactionLogAbortMarkers. Markers do not solve ordinary concurrent moves. Round22/23 regressions remain issue-linked skips; the explicit legacy diagnostic reproduces the owner wipe on 2.12.0. No serialization or compensating re-save is introduced.

Architecture alternatives and framing

Application callbacks cannot enforce placement on every thread. Recomputing ownership when configuration changes would move data without a migration. The installed map stays authoritative; slot moves and incarnation-based recovery remain separate work.

Framing-Verdict: chosen-approach-sound for the original architecture and core closing admission gate (17fab807b032). The final Pro planning reviewer proposed further failed-source inventory optimization (9550f6f252c4, better-alternative-exists). The implementation adopts bounded blob settlement, watchdog renewal and owned-only failure tracking. A capped per-key failure inventory is deferred: only an actual owned failure invalidates the scan; successful live traffic retains the completed proof. This author scope ruling is distinct from reviewer clearance. Targeted implementation review and its limitations are recorded below.

Retained contracts and review rulings

🤔 Complete shared history

A sharded database must retain a complete origin prefix, including ordinary-table transactions. A permanent apply error holds delivery until repaired. Missing frozen starts refuse the whole stream; local acknowledgments still mean local durability. Preserve unique ordinary writes before replacing the root. Arbitrary stale backups and new-root inheritance remain outside the approved recovery contract.

🤔 Evidence and activation fences

Cache only complete negative prior-origin evidence. An unavailable member keeps admission closed and triggers bounded retries. Existing-owner certification keeps its map fence while collecting proof; moving proof outside it would need another authority fence. This cold recovery cost is accepted to prevent a known replacement from certifying itself.

🤔 Reconnect availability

A below-issuance peer cursor waits briefly for older reservations, then refuses only that subscription until the closed floor catches up. Independent writers remain ready. A genuinely future cursor retains the durable recovery hold; clock skew or an authenticated bad cursor can prolong it. The highest committed key still cannot prove an earlier prefix closed.

🤔 Projection restart and point-read costs

Projection recovery deliberately restarts its non-owner walk; its separate completion marker never certifies owned origins. Sequence and frame commits omit redundant flushes; completion publication keeps its durability barrier. The unread projection cursor has been removed. Remote point reads also reconstruct frozen-start evidence before authorizing an owner response. #1121 records both costs; do not add a new resume protocol or stale-authority cache without measuring and preserving invalidation.

🤔 Test cleanup checks

Temporary roots pass through stopAndTeardownNodes to teardownHarper, which removes dataRootDir after stopping the node. Scalar configuration restores accept undefined. If a fixture’s initial module import fails, Mocha retains the original setup failure alongside any secondary teardown error.

🤔 Remaining review limits

Closed-continuation tests were checked by removing the actual guards: both mutants fail. Earlier broad reviews reported trace/input limits; the closing review is targeted to group (a) repairs. This is not fresh whole-feature coverage. Brief compatibility comments remain.

🤔 Placement overhead

Placement validation and per-peer encoding add work. Pro #1121 tracks measured and still-unmeasured optimization work; this PR makes no zero-overhead claim.

🤔 Review depth

Review limitations are detailed below. The owner limited this closing batch to introduced correctness and required invariants; the remaining root causes are filed below.

Closing review classification — correctness repairs and filed deferrals

Closing review classification

All original 61 core, 64 Pro and 17 docs findings and targeted additions are classified below. The 48 group(b) root-cause issues retain priority/effort/type/parent and Review-Deferral markers. Attribute moves are unavailable publicly and tracked in P1/v5.4 #1178. Only targeted delta reviews were requested.

Repository / root cause	Group and disposition	Evidence / rationale	Finding IDs
core / blob-error-taxonomy	(b) #3687 · P3 / Low	The current error already returns retryable 503; centralizing its superclass is taxonomy cleanup.	d08dacea0a75, e11f0afa6ffa
core / refusal-predicate-duplication	(b) #3688 · P3 / Low	Both adapter checks classify the current two errors identically; shared-helper cleanup.	c9a22b0acc08, 71fcfdf4e367, 5f72a22d756a
core / replica-audit-retention-cost	(b) #3689 · P3 / Medium	No measured harm; retain canonical blob references. Measure the replica scan separately from the ordinary early return.	231d22b5691a, f84015c7bbb6, f2476772f79e
core / unknown-attribute-slot-delete	(b) #3690 · P3 / High	No local row cannot prove remote absence. Route discovery/write forwarding is beyond the approved refusal contract.	5f9ab790c4da, 2ea50c90065d, 41608bda4551, 4a35b52a2ff2, ccf51ca668ea, 4399824535e6, 177f1e87c350, 65b8bafa4bd6, 8507e0aa571b
core / quarantine-error-detail	(b) #3691 · P2 / Medium	The write remains refused. Cached/raw-handle diagnostics can obscure the restart requirement; public status retains it.	7f700aa51b65, 9271cb3498d9
core / configuration-observer-failure	(a) implemented; regression covered	Initial readiness rejection removes change/error/close listeners. Reattachment registers one set. Refusal tests and actual restart prove unsupported policies do not poison healthy databases.	d31b8a6bf7a4, 928590eeefc5, 6564f91f74ad, 8c031a860027, 7e2708d53130
core / any-affinity-documentation	(b) #3692 · P3 / Low	Any may hold valid scalar values; unsupported values already fail before submission. Clarify documentation rather than narrow the schema.	1f63404554ac, e1cc6a71e52f, ed7513b9e61b
core / failed-declaration-hold	(b) #3693 · P3 / High	An attempted declarative activation deliberately installs the hold. Automatic rollback of that authority is outside the approved recovery contract.	03931ac171b9, c28f8c0c1f48, d73aaea06456, e15f99beec15
core / retained-owner-confirmation	(b) #3694 · P3 / Medium	Retaining confirmation for the logged attempt is deliberate; reducing excess acknowledgement after supersession is a separate optimization.	d1bf6fa407a7, b0308353f80c, 464473449148
core / repeated-owner-resolution	(b) #3695 · P3 / Medium	Repeated guards and owner lookup have no measured harm. Optimize only with equivalent invalidation and admission.	de00e9453145, bc879b0fc7f2, 67ab738c8452, b0b2309370c6, 171c2e12c320
core / local-only-retry-base-move	(a) public path refused; experimental work tracked in Pro #1178	The supported primary-key hash cannot change for a fixed key. Schema admission refuses by/slotFrom before durable declaration, and persisted unsupported policies quarantine only their database. The private experiment requires the switch plus marker capability/configuration for actual attributes. Concurrent attribute-move design and retained retry edge cases remain in #1178; native probes are preserved and skipped by default. No claim that markers solve ordinary concurrency.	a2106a5273ba, 7a6175f56276, 3332e121bd29, 2502542a9c82, e3f5b1a168c2, 220bb4607cc4, 049e6b6f04a6, c617c2ebee4b, eaead4d3791c
core / apply-test-inner-exemptions	(b) #3696 · P3 / Low	Tests exercise real copy/replication call shapes; inner defensive clauses are unreachable after the outer exclusion. Additional synthetic mutation coverage is optional.	c87a1fc96024, dd1d061a2cdb, c43607abe67a, 0d6039d3b957, dba44b67b104
core / retained-retry-admission	(a) public path refused; experimental work tracked in Pro #1178	The supported primary-key hash cannot change for a fixed key. Schema admission refuses by/slotFrom before durable declaration, and persisted unsupported policies quarantine only their database. The private experiment requires the switch plus marker capability/configuration for actual attributes. Concurrent attribute-move design and retained retry edge cases remain in #1178; native probes are preserved and skipped by default. No claim that markers solve ordinary concurrency.	d7f9f5e17cd5, d7dff86b04f1
core / synchronous-optional-replica-result	(b) #3697 · P3 / Low	The Pro caller contains synchronous errors in its Promise executor after the owner read. Normalizing the private method contract is a separate API cleanup.	3aa663632940, ead3f513eb35, 5de8cd976ec7, 940a8bae0a19, ed5d02096333, 3a9255072048, 827efff12e79, ee89c62149c5, 890a280751b6
core / received-owner-set-fence	(b) #3698 · P3 / High	The complete owner-set fence deliberately holds the stream across membership changes. Checking local membership only is a separate optimization.	28c4c1137978, faa4f5d8edc5, 5f3a066ea90a, 56f44186d21c, 16e614891103
core / invalid-slot-before-marker-gate	(a) public path refused; experimental work tracked in Pro #1178	The supported primary-key hash cannot change for a fixed key. Schema admission refuses by/slotFrom before durable declaration, and persisted unsupported policies quarantine only their database. The private experiment requires the switch plus marker capability/configuration for actual attributes. Concurrent attribute-move design and retained retry edge cases remain in #1178; native probes are preserved and skipped by default. No claim that markers solve ordinary concurrency.	259f11e7ebf7, 46633ce1fa85, 34e2ed97f9ea
core / markerless-gap-scope	(a) public path refused; experimental work tracked in Pro #1178	The supported primary-key hash cannot change for a fixed key. Schema admission refuses by/slotFrom before durable declaration, and persisted unsupported policies quarantine only their database. The private experiment requires the switch plus marker capability/configuration for actual attributes. Concurrent attribute-move design and retained retry edge cases remain in #1178; native probes are preserved and skipped by default. No claim that markers solve ordinary concurrency.	362311b5b0c2
core / duplicate-read-lifetime-check	(b) #3699 · P3 / Medium	Two synchronous closure checks are not a measured performance regression; keep the existing read-lifetime invariant.	c92591f49d5a, d1473de22beb, 4f913bc24ef2
core / detached-fill-cancellation	(b) #3700 · P2 / Medium	Detached cache-fill cancellation is a separate lifecycle extension; late request reads already stop at the logical-transaction guard.	f03d892c931d
core / added-comment-narration	(b) #3701 · P3 / Low	Comment and test-diagnostic cleanup, without a behavior defect.	f24170806c6c, 2729b9d65fc8, 7413e4bf79b7, c65a0edf4bc8, 296fb5c5c49a
core / blob-claim-collision-error	(a) implemented; regression covered	The recorded complaint asks for retryable 503 instead of an unlabelled 500. Current BlobReclamationBusyError supplies 503, code and retryable without naming an unrelated file.	79b036c0e635, b806a7b670b6, 7ce810571c2d
core / malformed-slot-encoder-input	(b) #3702 · P3 / Low	Supported writers supply a slot whenever its internal flag is set. Rejecting hypothetical malformed private input is defensive validation.	b6d58bb58890
pro / unused-projection-cursor	(a) implemented; regression covered	No reader consumes shardingProjectionCursor. Remove its durable writes and stale documentation; preserve projection row flush and completion marker.	fe0666935cce, f1b7d8e8bff6, 7a4fc941e8b2, c1c75dae87bf
pro / optional-configuration-rejection	(b) #1150 · P3 / Low	Current core catches initialization failures; the claimed unhandled rejection is not reachable through this producer. Track any contract cleanup with the observer work.	21bfcb76aaee
pro / added-comment-narration	(b) #1151 · P3 / Low	Comment cleanup; no behavioral change needed.	2676a11a36f3, 40d95f432124, 37a329713c7e, 2474456aa33e, 4c5713349924
pro / projection-agreement-after-map-move	(b) #1152 · P3 / High	The example requires a slot-moving map update, explicitly refused in #980 and deferred to #1053. Capture rules for future migration belong there.	0921aec60494, ad082d4bf1f6
pro / peer-refusal-diagnostics	(b) #1153 · P3 / Low	The refusal is enforced; additional expected/announced peer names improve diagnosis only.	271be31344ab, 0c5bb3089eb2, 724aef02bb08, 617254a6e786
pro / review-trace-bookkeeping	(b) #1154 · P3 / Low	The current Pro trace tables address the older 18-site gap. Preserve honest review limits; no product change or extra full review.	edd6598980b4, 1e5a0de8ef0c, ac1cd68a3936, 903679324d6f, 5ec511d08c03, 1ff8b5916ebb
pro / incomplete-evidence-retry-cost	(b) #1155 · P3 / Medium	Unavailable peers keep recovery closed; repeating successful evidence requests is an unmeasured cold-path cost.	e489942aa72a, dd866273d655, 3c8bc5a58907
pro / map-fence-proof-cost	(b) #1156 · P3 / Medium	The fence deliberately spans authority proof. Moving proof outside it requires a separate authority protocol; no measured harm supplied.	c676b787b0ab, 837d03c38113, af6dab02b66a
pro / bootstrap-shutdown-join	(b) #1157 · P2 / Medium	Joining/cancelling cold recovery work during shutdown is a lifecycle follow-up; no demonstrated live admission bypass in this finding.	3881670257c2
pro / non-owner-recipient-cost	(b) #1158 · P3 / Medium	Membership reads and recipient-array allocation are unmeasured performance work, not observed correctness failures.	0b0506e63f05, b0a8a01db1ed
pro / unbound-frame-table-scan	(b) #1159 · P3 / Medium	Per-frame table allocation on legacy unbound links has no measured harm. Cache invalidation needs its own measurement.	fecd77f1649c, c2d5c264fbd8, 3bfa2a5dfa91
pro / shared-stream-failure-hold	(b) #1160 · P3 / High	A database origin needs a complete prefix, including plain-table transactions. Skipping poisoned frames would violate the selected recovery contract.	a7c27fda57bb, 8304b7c6def7, 314b285ca385, 8dc6c2eac9ce, f67769120b98
pro / point-read-readiness-cost	(b) #1161 · P3 / Medium	Readiness authority is checked before serving owners; caching its catalog reads is unmeasured optimization work.	0f1febdbcc1e, 795551d183b0
pro / per-peer-wire-encoding-cost	(b) #1162 · P3 / Medium	Per-record reframing and allocation are unmeasured; retain capability and residency correctness.	0798040a08b3, 903b0d4daeb2
pro / mapped-frame-predecode-cost	(b) #1163 · P3 / Medium	Database-wide admission pre-scan adds decoding work; no measured harm or regression supplied.	64916b1c6e79, 1d23c2db74f8, 0115bd042ad6, 89ef798441ef
pro / certification-close-cleanup	(b) #1164 · P2 / Low	The rows are durable before publication and late callbacks cannot clear a replacement cursor. Notification/cleanup coalescing is a lifecycle follow-up.	88a12bcaf874, 40cf98321118, b3ad4eef21de
pro / authenticated-binding-contract	(b) #1165 · P3 / High	The normal producer is authenticated node metadata; an untrusted authorization-object producer is not established. Record boundary verification separately.	f2a2e2eca947, a643b6b77b5f
pro / copy-owned-blob-proof	(a) implemented; regression covered	Merged #1140 repair unchanged. Receiver refuses certification for PENDING owned bytes, including source ENOENT and checkpoint resume. A completed walk survives held blob-less and successful-blob live commits; a failed owned save behind the iterator invalidates its proof. Real receiver RED on 1b5016c, rebuilt GREEN; full integration in CI.	e85ee8f7c761, aa8246b72638, 153deeb4a98b, 6ebff25a9bd1, 8198df475e36, 515b35fe0778, c6e50ddfdaa5, bfb82d873f1c, 0cfe7f229e32, 801e846c7e20, 845b7f66194f
pro / repeated-certificate-revocation-cost	(b) #1166 · P3 / Medium	Per-damaged-row forced catalog flush is unmeasured copy-recovery overhead; no timing claim is made.	92851a9d6035
pro / unprobeable-blob-hold	(b) #1167 · P3 / High	Unknown completeness intentionally cannot certify. Broader repair support for non-probeable metadata is beyond this safety gate.	7989328a1464, d156359ae8b6, 2bee1bd6cfea, 2a36f8e47e6f, d2cd6905d973, 1a45579ad703
pro / upstream-copy-entry-read-cost	(b) #1168 · P3 / Medium	The one awaited stored-entry read per blob frame is merged #1140 behavior, explicitly retained by the owner.	6a333cf67182, c19df6872575
pro / blob-escalation-design-wording	(b) #1169 · P3 / Low	An upstream design paragraph says unconditional unlink although repair targets are preserved; documentation cleanup.	60302b3469f4, b25db39385e4
pro / copy-test-encoding-identity	(b) #1170 · P3 / Low	Decoded full body plus metadata is the semantic contract; harmless shared-structure re-encoding need not be byte-identical. Optional encoding coverage is separate.	94eb3020d6da
pro / missing-origin-plain-table-delivery	(b) #1171 · P3 / High	The approved complete-origin recovery contract refuses the shared stream until its start is proven; extending independent plain-table recovery is separate.	0aa847518e69, 236d2aca8282, 05fde5b867a8
pro / certificate-seqlock-documentation	(b) #1172 · P3 / Low	The slot allocations are corrected; documenting paired Int32 words is an additional design-note nit.	6023c031ab85
pro / negative-test-continuation-barrier	(b) #1173 · P3 / Low	The actual closed-continuation guard already has discriminating mutants. Additional synchronization in this negative helper is test hygiene.	ae8406f1b840, a7d8cf54bc2d
pro / confirmation-timeout-config-cost	(b) #1174 · P3 / Medium	Per-waiter config reads have no measured harm; optional cache invalidation optimization.	bbfe6d75bc77
pro / stale-backup-origin-detection	(b) #1175 · P3 / High	Arbitrarily stale same-name backups are excluded by the approved complete-original-name-backup requirement. Automatic detection is a recovery extension.	389141547e45
docs / new-table-downgrade-recipe	(b) #763 · P3 / High	The in-place recipe requires equivalent prior by-id placement. A new-table export/migration workflow is a separate capability.	f5e8a7cb4317, 349f87e33376, daa3ceb63491, 30fef8cf100b, 825cd30bcc74, 3d0079dda160, a4ceaf0e7935
docs / local-only-durability-warning	(b) #764 · P3 / Low	The guide already states owner delivery is absent and replicas expire/are omitted; a more explicit warning is documentation improvement, not a new refusal.	a59e3c86a674, 0b0002fca1d4
docs / local-only-caller-definition	(b) #765 · P3 / Low	Clarify the internal caller write option; do not invent an operations-API field.	a8c85b56b9b3, 5b8acf33e644
docs / marker-gap-operational-recovery	(b) #766 · P3 / Low	Exposure, release note and prevention guidance are now present. A tested post-incident repair procedure is separate; no compensating save is authorized.	ca3f0675072b, a00453f25016, 43a3811e8a6b, bffb3e786fbc
docs / binding-package-name	(b) #767 · P3 / Low	Name @harperfast/rocksdb-js to distinguish its 2.12.0 version from the engine version; naming nit.	12b155edfbae
docs / guide-slug-terminology	(b) #768 · P3 / Low	The existing residency guide route is valid; renaming it to match the public directive is editorial cleanup.	e2cd859b21ae, 68c9d5e35b37
pro / stale-slot-without-native-retry	(a) public path refused; experimental work tracked in Pro #1178	The supported primary-key hash cannot change for a fixed key. Schema admission refuses by/slotFrom before durable declaration, and persisted unsupported policies quarantine only their database. The private experiment requires the switch plus marker capability/configuration for actual attributes. Concurrent attribute-move design and retained retry edge cases remain in #1178; native probes are preserved and skipped by default. No claim that markers solve ordinary concurrency.
core / public-key-only-admission	(a) implemented; regression covered	Owner-directed closing scope: refuse by/slotFrom before durable side effects; enforce native marker prerequisite only under the private test switch; persisted policies use database quarantine. Key-derived concurrent PATCH controls retain full owners through reopen.	f66e1e4396a8, ef69bacff12e, e6d8d8a3bbd9, ac4238cd55d6, e08ed8068c6f
docs / key-only-sibling-scope	(a) implemented; prose checked	Removed the remaining pingTimeout and cluster-status attribute-move clauses, clarified confirmation fallback and put primary-key conversion eligibility before map installation.
pro / optional-lock-metadata	(a) implemented; npm regenerated lockfile	The final root lock metadata matches optional fulltext 0.7 and npm lock regeneration passes.	4826f528b8af

Targeted closing rulings

Finding IDs	Classification and evidence
f66e1e4396a8, ef69bacff12e	Intentional explicit-schema refusal required by owner: an unsupported declaration is rejected even when its persisted database is quarantined. Catalog recovery leaves other databases usable; an invalid mixed schema file still fails component loading. This private-switch transition is not supported public key-only behavior; Pro#1178 owns experimental enablement.
e6d8d8a3bbd9, ac4238cd55d6	Refuted by the native accessor contract: usesAbortMarkers returns false until this transaction logs a batch, so a throwaway unlogged transaction cannot establish capability. Owner specified capability plus enabled configuration at schema load, then per-transaction truth on retry. Experimental moves stay gated under Pro#1178 regardless of markers.
220bb4607cc4, 049e6b6f04a6	Fixed marker-capable test to require retryable409 with table/id, one submission, aborted marked handle, and unchanged concurrent row/indexes. Four cases pass with a stubbed true marker signal on2.12; this is branch verification, not native marker safety. Normal cases require real capability.
c617c2ebee4b, eaead4d3791c	Public attribute/expression declarations refused; key-derived placement cannot move for this id. Preserved experimental concurrency defects are tracked in Pro#1178, including absent-base/markerless variants; no sender/receiver redesign in closing scope.
c65a0edf4bc8, 296fb5c5c49a	Comment/restart-note cleanup remains group(b), core#3701. Closed shared watcher errors are observable and declaration retries refuse; no additional runtime change is required.
4f913bc24ef2	Group(b), core#3699: duplicated read check is unmeasured performance/cleanup, no demonstrated correctness failure.
e08ed8068c6f	Refuted: Pro replicationConnection.ts imports onShardingConfigurationChange and calls it at5363; Pro shardingBootstrap.ts imports setResidencyBaselineGuard and registers it at899. Companion consumers are present in #1116.
dba44b67b104	Existing group(b), core#3696. Public key-derived paths unaffected; experimental internal-apply extension remains in preserved tests.
e15f99beec15	Existing group(b), core#3693. Failed activation authority intentionally holds until repair/restart; unsupported public policies are refused before markers.
8507e0aa571b	Experimental-only, public policy refused. Existing group(b), core#3690, with concurrent-move design Pro#1178.
890a280751b6	Existing group(b), core#3697; optional cache failure is contained by Pro Promise executor after owner read resolution.
16e614891103	Existing group(b), core#3698; conservative canonical owner-set fencing is deliberate.
7f700aa51b65	Existing group(b), core#3691; writes still refuse, raw-handle diagnostics are separate.
7ce810571c2d	Correctness portion fixed: collision returns retryable503 without blaming an unrelated file; regression already RED/GREEN. Remaining unmeasured collision-path cost is group(b) measurement under Pro#1121, not an introduced data-integrity failure.
6ebff25a9bd1	Resolved finished-walk discard: native held-commit regression passes without rewalking, with fixed blob settlement and owned-failure invalidation. This is distinct from an incomplete walk encountering a live PENDING row.
2a36f8e47e6f, d2cd6905d973, 1a45579ad703	Group(b), Pro#1167: distinguish active transfers during an incomplete walk and optimize bounded slow-blob waits. The completed-walk defect is fixed; the one-million-row/200-writes scenario was not measured or reproduced. Owner excludes unmeasured perf extensions. Pending owned data must continue withholding readiness; no new scan cursor/inventory in closing scope.
515b35fe0778, c6e50ddfdaa5	Fixed CI coverage: bootstrap/complete-backup/fresh-name/retired-origin and restart-quarantine fixtures now use supported key hashing with no experimental switch; selected ids preserve their required owners. Syntax and hash choices checked; full execution remains CI-only per owner.
bfb82d873f1c, 0cfe7f229e32	Fixed: failure ownership classification is caught, forces revalidation, then unregisters/settles/removes the blob normally. New native receiver test injects lookup failure after ENOENT: RED on b952a38, GREEN with no certificate and no in-flight leak.
801e846c7e20, 845b7f66194f	Fixed barriers: wait for iterator exhaustion before withheld-success assertion; wait for persisted incomplete-row recovery hold before failure assertion, plus in-flight cleanup check.
4c5713349924	Group(b), Pro#1151. User explicitly requires retaining round22/23 probe scripts, including diagnostic history; no fixture deletion or renaming in closing scope.
b3ad4eef21de	Existing group(b), Pro#1164. Certification is already durable; an interrupt preserves the resume cursor and restarts validation. Further cleanup changes are separate.
5ec511d08c03, 1ff8b5916ebb	Group(b), Pro#1154. Parent-owned trace bookkeeping includes newly merged main sites; author verified the delta. Owner forbids a new full review solely to discharge the tool flag.
4826f528b8af	The final root lock metadata matches optional fulltext 0.7 and npm lock regeneration passes.

Final delta test-coverage nit

Finding	Classification and evidence
Bootstrap refusal ids lost-asia / lost-europe	(b) Fixture hygiene under Pro#1151. Both hash to slot0; this loop tests a database-wide history-loss admission refusal before owner routing, not separate owner/forwarding paths. The reviewer correctly notes the redundant region loop; no supported correctness invariant depends on it. Key-owner coverage is provided by the other explicitly chosen ids.

Final outside-review claims about an uncaught optional watcher rejection and a live projection-cursor writer are refuted by core/config/shardingConfiguration.ts (the returned promise has a reporting catch) and the absence of any production shardingProjectionCursor writer/reader. Historical metadata already written by older branch snapshots is inert. The incomplete-walk and bounded-settlement follow-up is #1167, not the unrelated #1177/#1178 numbers in the Cursor text. These are author evidence rulings, not new owner decisions.

Closing-major rounds (e6981556, ccc86ed8)

Two full rounds (codex, gemini, cursor-composer, domain adjudication) resolved all four majors and the follow-up per-frame table-scan minor. Their other open findings are outside these hunks and are listed, not fixed, per the scope ruling: the owned-blob repair checkpoint that excludes the incomplete row (replicationConnection.ts:4303, major, earlier bab61b0023c6), baseline publication re-certifying a revoked owned copy (shardingBootstrap.ts:874, major), the generation stamp race (residencyTopology.ts:91), unbound user-database DB_SCHEMA filtering (:5920), lazy value decoding in the owned-blob walk (:4237), the recovery listener left after forced retirement (:5380), and fixture narration nits. The CLI's framing recheck, raised by the shardingBootstrap.ts major, was not run for the same reason.

CI policy repair

Finding	Classification and evidence
Storage-binding importer coverage for resources/residencyPolicy.ts	(a) Fixed: add the capability-check module beside resources/residencyMaps.ts in framing_paths. CI main Node26 passed on main but failed on188d7f557; the focused workflow-format suite is now49/49.

| Retained attribute-policy suite fails the public gate | (a) Test setup fixed on2728a8757: RED40 failures on66f17ee82; enable the private switch within the suite and skip actual attribute cases without native marker capability. Public/key checks run now:21 pass/41 pending in both modes;53 policy cases pass with a capability stub. |
| Replica wire and failed-startup fixtures lag core main contracts | (a) Tests fixed ondc9aa9ab: RED two selected failures on6dd2af0f;GREEN20 across both files. Certified fixture supplies full-write-head; uncertified reply is returned uncached. Live sender support stays in Pro#1177. Watcher rejection must detach all listeners. |

🔧 Changes

Product behavior, admission, and recovery walkthrough

Product and architecture tour

Map and policy agreement

Initialization and conversion operations install immutable maps, compare legacy placement, and capture explicit existing-owner assertions. Authenticated schema setup authorizes map/policy publication before records. Conflicts hold durable progress. Recovery capability level requires the current evidence format before sharded writers resume. Cluster diagnostics exposes missing agreement and recovery sources.

Map and policy agreement flow

The arrows show the admission order.

flowchart LR
    A[Authenticated peer] --> B[Map and policy agreement]
    B --> C[Baseline and origin admission]
    C --> D[Record placement]

Loading

Placement and compatible records

Slot references bind a slot to map version and digest; compatibility encoding strips native slot headers for earlier peers. Owner reads resolve current owners. Confirmation deadlines bound explicit and automatic acknowledgment waits. Shared source authority publishes worker-visible state. Within the unsupported private attribute experiment, application localOnly moves away from the writer are refused before native submission because they cannot deliver the body. Other local-only writes need no confirmation; internal copy/replication apply remains exempt. Replicable non-owner attribute transitions require a connected new owner and durable acknowledgment; a failure after commit can be an uncertain outcome. Earlier-peer delivery support does not authorize activation with an incompatible sharding writer. The core companion supplies policy, storage, and transaction admission.

Baseline, origin history, and recovery

Bootstrap coordinator establishes a durable all-member initial witness before enabling origins. Private bootstrap RPC runs over authenticated member connections. Frozen starts bind each native root to its originating name; inheritance cannot borrow another root's start. Shared issuance holds records copy/resume bounds before writes and prevents stale worker readiness. Source certificates require complete owned data and a durable frozen start, independent of a temporary clock hold.

Lost-history diagnosis refuses a reused name with prior admission evidence. Status names complete-backup/new-name remedies, and source subscriptions close before retention can force a copy loop. Complete stopped-node backups preserve the original audit; arbitrary stale backups do not satisfy that contract. New-name admission needs explicit negative prior-origin evidence from current members. Co-owner failover includes unavailable origins on existing co-owner streams. Removing the old node does not discard retained audit logs. Whole-shard loss needs a complete backup. Incarnation-based identity remains #1123.

Copy and replay boundaries

Subscription replacement gives each subscription its own replay buffers and progress. Owned copies require a certified same-shard source, and keep a durable pending marker until certified, so an interrupted copy, even an empty one, is requested again. Projection copies have separate completion state and never credit owned progress. Snapshot rows cannot certify per-origin maxima; native copy anchors use the physical source log. Self-origin copy rows preserve issuance bounds before application. The baseline sentinel drains preceding writes before its owned-blob walk. A completed clean walk waits for a fixed blob set and survives subsequent successful live work; cursor removal still waits for drain. A replacement pass invalidates old callbacks before clone revocation can yield. Initial inbound handshakes persist authenticated frozen starts before setup continues. The replication design and root design index record the invariants.

Native topology and recovery evidence

Four-node placement tests cover operations/pool parity, key hashing, cold reads, PATCH/DELETE, and topology disagreement. Private attribute fixtures require native markers and skip on 2.12.0. Their component configuration, schema, local evidence resources expose actual stored state.

Recovery tests exercise fresh bootstrap, co-owner failover, visible source loss, name reuse refusal, old/new-origin convergence after rename, and complete-backup restoration with pool off/on. Quarantine tests check both declaration orders and repair/restart behavior using the component config, root-level evidence.

Conversion tests assert every existing owner's completeness before waiting for sharded replication. They verify no movement, missing/mismatched assertions, and optional previous-binary downgrade/re-upgrade. Conversion configuration, legacy schema, legacy placement and inspection reproduce last-digit placement. Declarative activation is RocksDB-only.

Focused boundary tests

The focused tests cover negotiation, real socket admission and blob routing, schema authorization, slot encoding, uncertified replica answers, configuration, bounded RPC, durable bootstrap transitions, worker issuance holds, native reservations and reconnect admission, name/start evidence, source eligibility.

Replay and confirmation regressions exercise owner acknowledgment, failed frame holds, origin cursors, send-loop scheduling, RPC reply failure, cancelled and replaced subscriptions.

Interrupted copies and rolling declarations

Copy repair uses merged #1140's shared resolver unchanged. The sharded readiness gate checks owned stored blobs after durable copy completion, including rows skipped by a resumed cursor and repairs whose source returns ENOENT. A hold revokes readiness and retains the copy cursor; ordinary completion or repair rechecks the stored row before certification. The check runs outside the commit callback and generic bootstrap certificate helper. A completed walk waits for a fixed set of in-flight blobs, while newer blob-less or successful-blob commits no longer discard it. An owned source failure behind the iterator invalidates the proof; cursor removal retains its zero-work gates. The unused projection cursor is removed, preserving projection flush and completion state. Declaration agreement lets a local directive resolve the peer-policy cycle while map/config checks and live admission remain enforced. Fresh-name recovery verifies owned blob bytes immediately after readiness.

Integrated core dependency

The full-text dependency and locked platform packages use 0.7.0, matching the integrated core's required binding contract. Pro's release bundle reads this lockfile. RocksDB remains pinned to 2.12.0; abort-marker enablement remains separate.

Retry and QA regression fixtures

integrationTests/cluster/experimentalSharding.mjs, integrationTests/cluster/fixture-sharding-rolling/config.yaml, integrationTests/cluster/fixture-sharding-rolling/resources.js, integrationTests/cluster/fixture-sharding-rolling/schema.graphql, integrationTests/cluster/shardingRollingDeclaration.test.mjs, unitTests/replication/concurrentSlotMoves.test.mjs, unitTests/replication/fixtures/concurrentMoves/round22-stub-fallback-hook.cjs, unitTests/replication/fixtures/concurrentMoves/round22-stub-native-helper.cjs, unitTests/replication/fixtures/concurrentMoves/round22-stub-replication.cjs, unitTests/replication/fixtures/concurrentMoves/round23-no-retry-native-helper.cjs, unitTests/replication/fixtures/concurrentMoves/round23-no-retry-replication.cjs, unitTests/replication/fixtures/slotRetryReplication.cjs, unitTests/replication/shardingAbortMarkers.test.mjs. These cover marker-gated abort/replay/replication, interrupted cursorless copies, and per-node declaration reloads. Native marker probes skip unsupported bindings; the explicit legacy-failure mode is intentionally red on 2.12.0.

✅ Verification

The closing-major fixes are at ccc86ed8: unitTests/replication passes 1621 with 14 pending, and the replica-copy check asserts byte equality with the demoted stub again. On an unsharded database with 100 tables, the per-frame gate drops from a 1,115 ns map read to a 5.4 ns cached check (turbo off, median of five 1M-call batches). Pro main, including #1177's fullWriteHead, is merged in at 1ce9bab4. core now points at core #3527's merge on harper main (b29241296). Locally on 1ce9bab4, residencyReplicas passes 9/9 and declarativeResidency passes 7 with 5 issue-linked skips. Those are the two suites that timed out on Node 26 cluster shards 5/6 and 1/6.

🤔 Closing review's four majors: fixed here

Fixed per Kris's ruling; each regression fails on 1ce9bab4:

• Unsharded databases skip the per-frame shard-map read through a cached verdict.
• An owned-copy pending marker lasts until certification, so an interrupted empty copy recopies.
• Recovery bounds wait for send authorization.
• Refused-generation records are skipped, not held.

Earlier published-head CI and merge order

At dc9aa9ab, cluster shard 1/6 and shard 5/6 timed out waiting for replicas. Core requires fullWriteHead === stub.version, and that Pro responder did not send the field. #1177 supplies it, and it is now merged in.

The earlier owner-approved receipt exception for dc9aa9ab is retired: head ccc86ed8 has its own full-round receipt.

Merge order: core #3527 and Pro #1177 are merged. This PR and documentation #753 stay draft until promotion.

Build, typecheck, required lint and targeted functional checks pass. Full integration is delegated to CI.

RED→GREEN evidence, native limits and QA

Build, typecheck and required lint pass. The post-main targeted batch passes 138 tests with 11 pending. The new round14 receiver regression fails on 1b5016ce, then passes for blob-less traffic and successful blob delivery while the live native commit is held; the certificate appears without discarding the completed walk, and the persisted copy cursor remains until commit drain. A source-ENOENT behind the iterator still withholds certification. The ownership-lookup failure regression is RED on b952a389 and GREEN after cleanup is protected; the blob settles without an in-flight leak. Negative assertions now wait for walk completion or the persisted recovery hold. Supported bootstrap and restart-quarantine integration fixtures use verified key hashes without experimental skips; CI will execute these full scenarios.

Merged #1140's copy repair is unchanged. F-1020 tests cover the interrupted first copy, source-ENOENT, checkpoint reconnect and persisted cursor; the published 323bf0c failed the rebuilt repair/readiness cases. F-1021 covers B-first/A-second live declaration. The current key-derived round23 controls preserve the complete owner through real RocksDB sender/receiver and independent reopen. Attribute round22/23 probes remain issue-linked skips; HARPER_TEST_ATTRIBUTE_MOVE_LEGACY_RED=1 intentionally reproduces both ordinary-concurrency owner wipes on 2.12.0.

CI fixture corrections are RED on 6dd2af0f (two selected failures) and GREEN on dc9aa9ab (20 tests across the replica-wire and watcher files). Synthetic certified replies now include the full-write-head required by core main. A new negative case proves an uncertified reply returns its body without caching; the live sender field now comes from merged Pro #1177. Failed-startup tests now assert listener detachment.

Fulltext 0.7.0 matches current core; RocksDB remains 2.12.0. Full integration runs in CI on the pushed heads. No benchmark lock or full local integration suite was used for this closing batch. Optional strict whole-tree lint has pre-existing warnings; required lint and changed-file formatting pass.

Final runtime targeted review: Claude, Gemini, Cursor Composer and domain adjudication completed on 95cd9f32; the cleanup, barriers, CI skips and lockfile findings were resolved. The redundant region loop is fixture hygiene under #1151; it exercises a database-wide refusal before owner routing. The broader incomplete-walk proposal stays author-classified group(b) under #1167. Subsequent core pins and replica/watcher test corrections have no new independent receipt; production code is unchanged. The tool requests a closing full round; the owner explicitly forbids one, so none will run. Earlier trace bookkeeping is retained, not claimed discharged.

PR QA: Does a late shard owner killed during its empty owned copy's certification come back certified? — queued @ ccc86ed, qa-pr-harper-pro-1116-ccc86ed

_{🤖 Generated by OpenAI Codex; posted via @kriszyp.}

Release target: v5.4 — new declarative cluster-placement surface for the next minor release

Complexity: complicated

Related PRs: #554 overlaps, #717 overlaps, #995 overlaps, #1054 overlaps, #1092 overlaps, #1131 overlaps, #815 overlaps, #1101 overlaps, #1112 overlaps, #1133 overlaps, #1140 overlaps, #1177 overlaps

Review deferral receipts

Deferred findings

• 1d1518092d8c3710 major performance — replication/replicationConnection.ts — Residency heartbeat affects ordinary sending subscriptions — filed Reduce repeated declarative placement checks and sender bookkeeping #1121
• 72a06948797a6e07 major reliability — replication/replicationConnection.ts — Incomplete owned-blob walks can repeatedly hold or reconnect — NOT FILED
• c07750f66d5120fd major performance — replication/knownNodes.ts — Confirmation timeout cache retains per-waiter configuration reads — filed Measure configuration reads per confirmation waiter #1174
• 0a7544e3e122413c major performance — replication/residencyTopology.ts — Non-owner writes rebuild replication recipients — filed Measure recipient lookup and allocation on non-owner sharded writes #1158
• 2fda9a27a323578b major performance — replication/replicationConnection.ts — Mapped database frames incur a second decode pass — filed Measure the admission predecode pass on mapped database frames #1163
• 6eb764a1e8fde5c6 major performance — replication/replicationConnection.ts — Unbound frames allocate database table scans — filed Measure per-frame table discovery on unbound legacy replication links #1159
• dc9d1534e325351c major performance — replication/replicationConnection.ts — Sharded point reads reconstruct bootstrap readiness — filed Measure bootstrap catalog reads on sharded GET_RECORD requests #1161
• c638e978c47969b3 major performance — replication/residencyWire.ts — Per-record per-peer reference construction and header encoding — filed Reduce repeated declarative placement checks and sender bookkeeping #1121
• fa1239694ff06b06 major reliability — replication/replicationConnection.ts — Database-wide permanent failure hold wedges unrelated tables — filed Define recovery options for a poisoned plain-table frame in sharded history #1160
• 9379f0883f02d130 major reliability — replication/replicationConnection.ts — Missing sharding origin start stops ordinary-table delivery — filed Define independent plain-table recovery when a sharded origin start is missing #1171
• +20 more in the review receipt

🤔 defer this PR's major performance finding 1d1518092d8c3710 (Residency heartbeat affects ordinary sending subscriptions) to #1121 instead of fixing it here?

🤔 defer this PR's major reliability finding 72a06948797a6e07 (Incomplete owned-blob walks can repeatedly hold or reconnect) to a separate issue instead of fixing it here?

🤔 defer this PR's major performance finding c07750f66d5120fd (Confirmation timeout cache retains per-waiter configuration reads) to #1174 instead of fixing it here?

🤔 defer this PR's major performance finding 0a7544e3e122413c (Non-owner writes rebuild replication recipients) to #1158 instead of fixing it here?

🤔 defer this PR's major performance finding 2fda9a27a323578b (Mapped database frames incur a second decode pass) to #1163 instead of fixing it here?

🤔 defer this PR's major performance finding 6eb764a1e8fde5c6 (Unbound frames allocate database table scans) to #1159 instead of fixing it here?

🤔 defer this PR's major performance finding dc9d1534e325351c (Sharded point reads reconstruct bootstrap readiness) to #1161 instead of fixing it here?

🤔 defer this PR's major performance finding c638e978c47969b3 (Per-record per-peer reference construction and header encoding) to #1121 instead of fixing it here?

🤔 defer this PR's major reliability finding fa1239694ff06b06 (Database-wide permanent failure hold wedges unrelated tables) to #1160 instead of fixing it here?

🤔 defer this PR's major reliability finding 9379f0883f02d130 (Missing sharding origin start stops ordinary-table delivery) to #1171 instead of fixing it here?

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini,codex,cursor-grok; late=cursor-grok:adjudicated; adjudicated=domain; declined=muse; rounds=19; full=10 @ ccc86ed}

_{Review-Attention: deep ~375m (critical: clusterStatus.ts, knownNodes.ts +12; sensitive: package-lock.json; decisions: do-less-alternative, attribute-slots-behind-test-switch, whole-stream-hold-on-placement-error, sharding-confirmation-timeout-30s, operator-assertion-as-trust, deferred-1d151809, deferred-72a06948, deferred-c07750f6, deferred-0a7544e3, deferred-2fda9a27, deferred-6eb764a1, deferred-dc9d1534, deferred-c638e978, deferred-fa123969, deferred-9379f088; raised: degraded review) @ ccc86ed}

[gemini-code-assist]

Code Review

This pull request implements declarative residency agreement and wire references for replication, introducing topology agreement checks, slot reference resolution, and wire encoding reframing, alongside comprehensive integration and unit tests. The review feedback focuses on improving the robustness of these new tests by avoiding optional chaining in assertions and explicitly verifying that parent objects exist before accessing their properties to prevent vacuous passes or unhandled TypeErrors.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​harperfast/​fulltext@​0.7.0

View full report