Skip to content

fix(dst): poll operator bodies on the seeded runtime under DST - #1174

Closed
ragnorc wants to merge 2 commits into
ModernRelay:mainfrom
ragnorc:fix/load-holders-sized-by-the-producer
Closed

ragnorc wants to merge 2 commits into
ModernRelay:mainfrom
ragnorc:fix/load-holders-sized-by-the-producer

Conversation

@ragnorc

@ragnorc ragnorc commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1160. Since #1158, GQT (dst) fails intermittently with replay_mismatch. Two replays of one seed report different actual sizes for the same query_memory_bytes refusal: 26677, 26285 and 26349 across 60 runs of load_csv_integer_endpoint_memory at seed 0, with 16 of 60 mismatching at da1c4636. Every graph operator polls its body through WorkMemory::blocking_body, which runs each poll in tokio::task::spawn_blocking. DST seeds a current-thread runtime, but a blocking thread is outside it. A csv load's producer and its ValidateExec sweep therefore charge and release one memory pool from two OS threads, and a refusal reports whatever the pool held at an instant the seed does not order. This is the class #1161 describes, and this change removes it at that seam rather than at one operator, replacing this PR's first approach (sizing unique holders on the producer), which the review showed did not touch the edge load.

omnigraph_core::dst_blocking is a new thread-local seam beside dst_gate, dst_clock and dst_ids. Without the dst feature its on_runtime() inlines to false, so production polls exactly as before. The DST universe (omnigraph_dst::run_universe, the path GQT's DST runner takes) installs OnSeededRuntime on its harness thread next to the logical clock and seeded IDs. When it is installed, blocking_body polls the body as one of the seeded runtime's own tasks. It keeps the cancellation check before every poll, the compute timer, the blocking-work probe and the cancellation disarm. It installs no worker checkpoint hook, because that hook is keyed by thread and every body now shares one; checkpoint() then only checks, as it does with no probe installed. spawn_blocking remains the production path and the only one outside DST: no other engine source calls it, so NeedleRows::build, the scan runtime filter and every other operator body are covered by the same branch. Lance's own CPU pool (spawn_cpu) stays outside the seed, as before; DST already runs it as one FIFO worker under LANCE_CPU_THREADS=1.

blocking_bodies_run_on_the_seeded_runtime_under_the_dst_seam_issue_1160 pins the mechanism: without the seam a body runs on a blocking thread, and with it the body, and a body it nests, run on the installing thread. Locally, at this head, 60 runs of omnigraph-gqt load_csv_integer_endpoint_memory.gqt --target omnigraph-engine-dst --storage in-memory-object-store --seed 0: 0 replay mismatches among the 33 runs that completed. At the review's measured rate on main (about 27% of runs mismatch), seeing none in 33 has a probability of about 3 in 100,000. The other 27 exceeded the case's 10-second budget on a host shared with other sessions' builds and GQT runs (load average 10 to 13 during the loop). Later, at a load average of about 50, the same case on the plain omnigraph-engine target, which this change does not reach, timed out the same way, so the timeouts are the host's and CI's GQT jobs are the timing evidence. The DST pinned suite passes, including the storage-cost golden, with no regenerated line. Both clippy shapes pass on the pinned 1.97.1 toolchain: the shipped engine without dst, and the engine with dst plus omnigraph-dst.

For review: the decision to challenge is running every operator body on the one DST thread. A body that blocked its thread synchronously would now stall the universe instead of one blocking worker; I found no such wait in engine sources outside tests. Unquiesced fleet runs lose engine-level body overlap too, which they made no replay claim about; they keep Lance's thread overlap. The concurrent harness's actor threads (omnigraph-dst/src/concurrent.rs) install their own slots and do not install this one, so their bodies still run on blocking threads; extending the seam there is a separate decision for that harness. The write-load pair in #1160 (398742 and 399198) did not reproduce on the reviewer's host in 90 runs, so this PR claims only the edge load; it shares the same seam.

The sweep sized the holders with grow_if_free on the first batch it
received, while the csv producer, on its own blocking thread, was still
releasing that slice's charges. Whether the tables were charged depended
on that race, so a later query_memory_bytes refusal reported a different
actual in two replays of one seed. The source now runs the sizing on the
partition that publishes the row estimate, before that slice's batches
leave it.

ragnorc commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Recommendation: revise.

A DST replay of one seed must report one history. Issue #1160 shows two query_memory_bytes actual values for one seed on a csv load. The parser runs in WorkMemory::blocking_body, which calls tokio::task::spawn_blocking (crates/omnigraph/src/engine/operators/memory.rs). The DST seed does not order that thread. A pool sample taken while the thread still runs can differ on the next replay.

This pull request moves unique-holder sizing onto the csv partition. ValidateExec::execute installs an EstimateObserver before input.execute. The partition that wins first_slice_rows calls size_unique_holders before sender.send. The closure keeps Weak handles to the statement memory and the sweep plan, runs once, and is dropped.

Impact

load_csv_integer_endpoint_memory still mismatches at this head. The same three actuals appear at the merge base. Revise the change. Put the replay schedule in blocking_body, the seam #1161 names. Remove the observer.

Blocking

  1. The edge load still has more than one actual.
    At da1c46368c33876d61615add0ceb6778f9ce2b58, seed 0, 60 runs: 16 replay_mismatch. The actuals are 26677, 26285, and 26349.
    At 21a5b9c4b4f25a4979b843866c64f8919fe66642, seed 0, 60 runs: 23 replay_mismatch. The actuals are the same three values.
    The gaps are 392 bytes and 64 bytes. One holder table for one shard, with fewer than 8 rows in the shard, is 8 × (size_of::<(Vec<String>, (String, u64))>() + 1) = 456 bytes. I measured that size on this toolchain.
    The plan for this load is two edge_ri checks and one partition (load_csv_integer_endpoint_memory.gqt). SweepPlan::new builds holders only for ValidateCheck::Unique. This load has an empty holder list, so size_unique_holders performs no grow_if_free. The move leaves this actual on the same race.
    On both commits, source_bytes.shrink_to and drop(decoding) run before first_slice_rows.set and before sender.send. The sweep sees the batch after those two releases. The producer can still drop the source bytes and its WorkMemory after the send, on the blocking thread.

  2. The schedule owner is blocking_body.
    spawn_blocking stays in place. input.execute starts the producer, and the held body then runs admit_sets, so set admission still overlaps the producer thread. NeedleRows::build and the scan runtime filter still call grow_if_free off the seeded runtime. A csv load with several partitions still overlaps its producers. The pull request text says that.
    EstimateObserver adds a mutex slot, a FnOnce, a one-shot take, and a Weak upgrade that returns Ok(()). The next operator with the same overlap needs another callback. feat(dst): run the engine's blocking operator work on the seeded runtime #1161 covers the class in one seam. This pull request leaves that issue open, and it leaves bug: DST replay mismatch, a load's memory refusal reports a timing-dependent actual #1160 open for the edge load.

Optional

  1. ValidateExec ignores the bool from observe_rows_estimate. A failed Weak upgrade returns Ok(()) and skips sizing with no trace. During a live execute the strong refs are on the stack and then in the held stream, so the skip is the cancelled case. grow_if_free is best-effort there. The same Ok(()) also hides a Weak that can never upgrade.

  2. The producer takes the observer only when rows_estimate() is Some. This producer awaits resolved() before the slice loop, so the estimate is Some after a successful set. A later edit that sets the lock first would leave the callback in the slot, and no later batch would retry it. After take, the slot is empty, so a later observe_rows_estimate returns true and stores a callback the OnceLock winner will not run.

  3. the_row_estimate_reaches_its_observer_before_the_first_batch_issue_1160 records the callback in a private Vec. It shares no pool with a consumer. It passes while the GQT replay fails. The existing GQT case is the replay proof.

  4. The charge parent moves from the sweep facts child to the ValidateExec memory. Both use the statement pool. release_holders still drops the charge boxes. The pull request states this.

Tradeoff and liability

grow_if_free is best-effort. A missing holder reservation lets the map grow as it fills, so the uniqueness rows stay correct. The value that must replay is the refusal actual. SpillHeadroomPool::try_grow writes last_refusal, and classify_failure reports reserved + additional from that record. WorkMemory::grow_if_free returns Ok(false) on a decline and leaves the record in place. A later ResourcesExhausted from grow reports it. Sampling reserved on a thread the seed does not order makes actual a race.

The new callback pins one optional charge to the producer. That charge is empty for the edge load that fails here. The write-load pair in #1160 differs by 456 bytes (399198 − 398742), which matches one holder table. This host ran write_max_bytes_removed_id_charge at both commits and saw only actual 399198. Use the issue for that pair. Use these runs for the edge load.

Five more operators would add five observers. One DST-selected branch inside blocking_body schedules every blocking body on the seeded runtime. Production keeps spawn_blocking. DST then runs those bodies one schedule. RFC 0037 hunts real overlap in unquiesced runs, so that cost belongs on the DST seam. The per-site callback changes production order to chase one DST number, and it misses the number in the case that fails here.

Pinned Lance 13.0.0 spawn_cpu (lance-core-13.0.0/src/utils/tokio.rs) calls spawn_blocking on its own runtime. DST sets LANCE_CPU_THREADS=1, which uses one FIFO worker. Those jobs stay off the seeded runtime. This pull request does not touch Lance, the OpenAPI document, or the DST storage-cost golden. I see no open pull request that implements #1161.

Validation

Host: Linux 6.12.94+, x86_64, 4 CPUs, rustc 1.97.1. Debug omnigraph-gqt. Three busy Python processes during each loop. Each invocation replays the seed twice:

omnigraph-gqt <case> --target omnigraph-engine-dst --storage in-memory-object-store --seed N --artifacts <dir>

Merge base da1c46368c33876d61615add0ceb6778f9ce2b58. Binary built in /workspace with cargo build -p omnigraph-gqt --locked --bin omnigraph-gqt. The loop invoked /workspace/target/debug/omnigraph-gqt. That path was removed afterward. The preserved copy /tmp/og-review/omnigraph-gqt-main (05:35, the loop started 05:36) has no observe_rows_estimate symbol.

  • load_csv_integer_endpoint_memory seed 0 × 60: 16 mismatch, 0 other, 56.6s. Pairs (replay 0, replay 1): 42× (26677, 26677), 9× (26677, 26285), 5× (26285, 26677), 2× (26285, 26285), 1× (26285, 26349), 1× (26349, 26677).
  • write_max_bytes_removed_id_charge seed 0 × 30: 0 mismatch, actual 399198, 31.8s. Seed 42 × 30: 0 mismatch, actual 399198, 31.0s.
  • Same case, seed 0, 4 waves × 8 parallel invocations: 0 mismatch, actual 399198, 12.2s.

Head 21a5b9c4b4f25a4979b843866c64f8919fe66642, worktree /tmp/og-review/head, CARGO_TARGET_DIR=/tmp/og-review/target-head cargo build -p omnigraph-gqt --locked --bin omnigraph-gqt. The binary contains observe_rows_estimate.

  • load_csv_integer_endpoint_memory seed 0 × 60: 23 mismatch, 0 other, 63.1s. Pairs: 34× (26677, 26677), 12× (26285, 26677), 9× (26677, 26285), 3× (26285, 26285), 1× (26349, 26677), 1× (26677, 26349).
  • write_max_bytes_removed_id_charge seed 0 × 30: 0 mismatch, actual 399198, 34.6s. Seed 42 × 30: 0 mismatch, actual 399198, 34.1s.

Every attempt outcome was Ok. The 25600 step refused and the 32768 step admitted on every run. The failure is the cross-replay compare of actual.

Unit tests, --exact --test-threads=1, 1 passed each:

  • Merge base, /workspace/target/debug/deps/omnigraph-88a9de5af52995a7: engine::operators::csv_source::tests::the_row_estimate_precedes_the_first_batch ok; engine::operators::validate::tests::held_batches_are_probed_once_the_sets_arrive ok.
  • Head, /tmp/og-review/target-head/debug/deps/omnigraph-88a9de5af52995a7 (test profile, 15m 14s): engine::operators::csv_source::tests::the_row_estimate_reaches_its_observer_before_the_first_batch_issue_1160 ok; engine::operators::validate::tests::held_batches_are_probed_once_the_sets_arrive ok.

Limitations

  • This host did not show 398742 against 399198 at either commit. Issue bug: DST replay mismatch, a load's memory refusal reports a timing-dependent actual #1160 is the evidence for that pair.
  • Debug build, 4 CPUs, and added CPU noise. The mismatch rate can move. The edge-case actual set stayed the same.
  • No run used a limit between 26285 and 26677. Both limits in the case sit on one side of all three actuals.
  • I read Lance spawn_cpu. I did not run a separate Lance-thread race.
  • These local runs are the proof. I did not use CI on this head.

Inline comment, crates/omnigraph/src/engine/operators/validate.rs line 162, RIGHT:
This callback is empty for the load that still mismatches. load_csv_integer_endpoint_memory plans two edge_ri checks and no ValidateCheck::Unique, so size_unique_holders charges nothing. input.execute below starts the producer, and admit_sets in the held body still overlaps that thread. The bool from this call is unused. A failed upgrade returns Ok(()) and skips the charge with no trace.

Inline comment, crates/omnigraph/src/engine/operators/csv_source.rs line 452, RIGHT:
shrink_to and drop(decoding) already ran above this set. The send is below it. The body runs on spawn_blocking, and the DST seed does not order that thread against the sweep. The take sits behind rows_estimate(). This producer awaits resolved() first, so the estimate is Some after set. A path that sets the lock while the estimate is still None leaves the observer in the slot, and no later batch retries it.

Inline comment, crates/omnigraph/src/engine/operators/csv_source.rs line 278, RIGHT:
A full slot returns false. After the winning partition takes the observer, the slot is empty, so a later call returns true and stores a callback that first_slice_rows will not run again.

WorkMemory::blocking_body polls every graph operator's body on a
spawn_blocking thread, which DST's seeded current-thread runtime does not
order. A csv load's producer and its sweep charged one memory pool from
two such threads, so a query_memory_bytes refusal reported a different
actual on two replays of one seed. A new thread-local seam,
omnigraph_core::dst_blocking, makes blocking_body poll the body as one of
the installing runtime's tasks; the DST universe installs it beside its
clock and IDs. Production keeps spawn_blocking.

This replaces sizing unique holders on the producing partition, which
left the edge load's race untouched.
@ragnorc ragnorc changed the title fix(load): size a csv load's unique holders on the producing partition fix(dst): poll operator bodies on the seeded runtime under DST Oct 27, 2026
@ragnorc

ragnorc commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Revised as asked, at 62285bf, and the description is rewritten.

Blocking 1 (the edge load keeps more than one actual). Agreed: the edge load has no unique holders, so the observer could not touch it. The observer is gone; csv_source.rs and validate.rs are back to main.

Blocking 2 (the schedule belongs in blocking_body). It is there now. omnigraph_core::dst_blocking is a thread-local seam beside dst_gate, dst_clock and dst_ids, and it inlines to false without the dst feature. run_universe installs OnSeededRuntime next to the clock and IDs. With the seam installed, blocking_body polls each body as one of the seeded runtime's tasks, keeping the cancellation check, the timer, the probe and the disarm. spawn_blocking stays the production path and is the engine's only blocking spawn, so NeedleRows::build, the scan runtime filter and multi-partition producers go through the same branch.

At this head, with your loop at seed 0 for 60 runs: 0 mismatches among the 33 runs that completed, against your measured 16 of 60 on main. The other 27 exceeded the 10-second budget on an overloaded shared host; the same case on the plain omnigraph-engine target, which this change does not reach, timed out the same way. The DST pinned suite passes with the cost golden unchanged, and both clippy shapes pass on 1.97.1.

Optional 1–4. All four concerned the observer. 1, 2 and 4 are moot now that it is removed. For 3, the mechanism test is replaced by blocking_bodies_run_on_the_seeded_runtime_under_the_dst_seam_issue_1160, and the GQT replay remains the symptom proof.

Left open, stated in the description. The concurrent harness's actor threads do not install the seam; extending it to them is a separate decision for that harness. And the write-load pair from #1160 is not claimed.

ragnorc commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Recommendation: revise.

A DST replay of one seed must report one query_memory_bytes actual. This head still reports two actuals for the edge load when other CPUs are busy. The new seam is the right place for the old spawn_blocking race. The new public re-export fails the engine fence.

Prior findings

Blocking 1 (the edge load keeps more than one actual): open. The actuals are the same pair. The path that produces them is new. See blocking finding 1.

Blocking 2 (the schedule belongs in blocking_body): addressed. The observer is gone. csv_source.rs and validate.rs match main. blocking_body polls inline when on_runtime() is true. run_universe installs OnSeededRuntime beside the clock and the ids.

Optional 1–4: addressed. Each one described the observer. The observer is removed.

The replacement test does not lock the actual. See optional finding 1.

Blocking

  1. The edge-load actual still changes across two replays of seed 0.

load_csv_integer_endpoint_memory plans two edge_ri checks and one partition. The case text says the refusal replays with one actual. comparable keeps the error text, so two actual values are a replay_mismatch.

Quiet loop, seed 0, no extra processes. Artifact timestamps:

  • Merge base da1c4636, 60 runs, span 57.9s: 3 replay_mismatch. Pairs: 57× (26677, 26677), 2× (26677, 26285), 1× (26349, 26677).
  • This head, 60 runs, span 59.7s: 0 replay_mismatch. Every pair is (26677, 26677).

That quiet result agrees with the reply (0 mismatches in 33 completed runs). The reply also reports 27 timeouts. A timeout is not a match. The loop that found this bug keeps three CPUs busy. The reply does not report that load.

Busy loop, seed 0, three python3 -c 'while True: pass' processes:

  • Merge base, 40 runs, span 38.3s: 15 replay_mismatch. Pairs: 25× (26677, 26677), 8× (26285, 26677), 7× (26677, 26285).
  • This head, 40 runs, span 38.2s: 21 replay_mismatch. Pairs: 15× (26677, 26677), 11× (26285, 26677), 10× (26677, 26285), 4× (26285, 26285).

Every attempt was Ok. The 25600 step refused. The 32768 step admitted. The failure is the compare. Both limits sit on one side of both actuals, so one GQT run passes when the two replays happen to match. GQT (dst) passed on this commit for that reason.

The seam does put every pool charge on the universe thread. One busy mismatch was traced with a temporary log (isolated worktree, edits restored, nothing pushed). All 1500 pool lines used ThreadId(2). All 64 blocking_body entries took the DST path. Zero took spawn_blocking. The two refusals are graph Arrow allocation of 1096 bytes, at reserved 25189 and reserved 25581.

25189 + 1096 = 26285. 25581 + 1096 = 26677. The gap is 392 bytes. Those bytes are load integer endpoint keys grows of 64, 200, and 128. One replay holds them before the Arrow grow. The other replay does the Arrow grow first.

ValidateExec::execute calls input.execute before admit_sets. input.execute spawns the CSV producer (produced_stream → RecordBatchReceiverStreamBuilder::spawn). admit_sets is the call that charges the endpoint keys, and the comment on that function says the charge precedes the drain. The producer can charge first, because it is already a runnable task. futures::join! then overlaps the drain with load_sets.

poll_fn polls the body on the caller. It does not start a separate task, and it does not fence a wake from another thread. Pinned Lance 13.0.0 spawn_cpu (lance-core-13.0.0/src/utils/tokio.rs) runs the closure on its own runtime (thread_name("lance-cpu"), worker_threads(1), max_blocking_threads from LANCE_CPU_THREADS). DST sets LANCE_CPU_THREADS=1. That completion wakes the OmniGraph runtime from outside the seed. This re-review did not turn that runtime off, so the wake source is the class "another thread". Rayon and the object store can wake the same queue.

dst_blocking says the seed orders every body's charges and that the refusal size is a function of the seed. The busy loop contradicts that sentence. Keep the seam. It is the right owner for "which thread polls an OmniGraph body", and it removes the two-thread pool race for every blocking caller. It is not the owner of outside wakes. Make this actual the same on every replay of seed 0 while other CPUs are busy. Do that at the pool-charge schedule. Do not add a CSV callback for these 392 bytes. The previous observer missed this load.

  1. The new public re-export is not on the engine fence.

crates/omnigraph/src/lib.rs has pub use omnigraph_core::dst_blocking when feature dst is set. On this commit, graph_manifest_writer_methods_are_not_public_escape_hatches failed in Test Workspace and in Storage Upgrade Compatibility. The panic text is:

lib.rs: pub use omnigraph_core::dst_blocking is not in SPLIT_CRATE_REEXPORTS; re-export it pub(crate), or pin the exact item after review; a crate or module export publishes every writer inside it

SPLIT_CRATE_REEXPORTS already pins lib.rs: omnigraph_core::dst_clock, dst_gate, and dst_ids. Pin lib.rs: omnigraph_core::dst_blocking in that list, or re-export the module pub(crate) and install it from omnigraph_core. CI Gate is red because those two jobs failed. Run 40477751365.

Optional

  1. blocking_bodies_run_on_the_seeded_runtime_under_the_dst_seam_issue_1160 checks the thread id of one body and of a nested body. The two calls do not charge one pool. The test does not compare actual. #[cfg(feature = "dst")] keeps it out of Test Workspace, which does not enable dst. cargo test -p omnigraph-dst does not run engine lib tests. The fix-regression gate accepts the function name. The gate does not run the test. One GQT invocation can pass when both replays match. This re-review did not run the unit test.

  2. The DST branch uses let _owner = owner and returns while work is still alive. QueryWorkLease::own drops the future before the lease, so settlement is announced after the work is gone. The production poll uses own. This branch drops the lease first. On a current-thread runtime the waiter cannot run until this task yields, so a replay does not show the early signal today. Hold the lease with own.

  3. The omnigraph-dst crate docs list dst_ids, dst_clock, and dst_gate. Add dst_blocking to that list.

Tradeoff and liability

The seam is one primitive beside the three existing seams. on_runtime inlines to false without feature dst. One branch covers every blocking caller: the CSV producer, NeedleRows::build, the scan filter, and a multi-partition load. That is less liability than one observer per operator. Five more operators do not need five more callbacks.

The branch is a second mode. It skips the checkpoint hook, because that hook can wait on a condvar and the universe thread is the only thread. It holds the blocking guard across awaits. A panic unwinds, and the production path maps a spawn_blocking panic to graph worker failed. Production does not install the seam, so the server stays on spawn_blocking. Those differences stay acceptable while only the harness thread installs the seam.

The seam does not own wakes from other runtimes. Lance 13.0.0 still has a CPU runtime. No open pull request replaces that runtime. A later Lance deterministic mode can move spawn_cpu onto the seeded thread. Until that lands, a sample of reserved during overlapping tasks stays exposed. The module comment currently states the stronger claim. Narrow the comment to the thread the body runs on.

The concurrent harness does not install the seam. That harness states no replay-identity claim. Leave that harness for a separate change.

write_max_bytes_removed_id_charge is not this proof. Seed 0, 20 runs, both commits, 0 mismatches. The actuals are 65560 and 399198. This host did not show 398742.

Validation

Host: Linux, 4 CPUs, rustc 1.97.1, debug omnigraph-gqt. Merge base da1c46368c33876d61615add0ceb6778f9ce2b58 (git merge-base of this head with origin/main at 790174c). Head 62285bfffb76653f9bb1dddc97d452a2e832b321.

Command, once per run:

omnigraph-gqt <case> --target omnigraph-engine-dst --storage in-memory-object-store --seed 0 --artifacts <dir>

The base binary has no dst_blocking symbol. The head binary contains omnigraph_core::dst_blocking::on_runtime. Counts and timestamp spans are above.

CI on this commit, run 40477751365: GQT (dst) pass, DST pinned suite pass, Lint (clippy) pass, Format (rustfmt) pass, Fix Regression Gate pass. Test Workspace fail and Storage Upgrade Compatibility fail on the forbidden-API test above. CI Gate fail names those two jobs. A green GQT (dst) job is one pair of replays. It is not the busy loop.

Local clippy was not run. The new unit test was not run.

Limitations

The busy sample is 40 runs. The mismatch count moved (15 at the base, 21 at this head). The actual set did not: 26677 and 26285. The quiet merge base also showed 26349 once. The head samples did not.

The wake source was not isolated by disabling Lance's CPU runtime.

No limit in the case sits between 26285 and 26677.

The charge trace was a temporary log. Those edits are restored. Nothing was pushed.

Recommendation: revise.

Inline, crates/omnigraph-core/src/dst_blocking.rs line 9, RIGHT:
Two replays of seed 0 still report actual 26677 and actual 26285 when three other CPUs are busy: 21 of 40 runs at this commit. The gap is 392 bytes of load integer endpoint keys (64 + 200 + 128) held on one replay when graph Arrow allocation grows by 1096. Every one of those charges was on the universe thread, and spawn_blocking was not taken. An outside wake still picks the order. Narrow this claim. Make the sample stable under that load.

Inline, crates/omnigraph/src/engine/operators/memory.rs line 364, RIGHT:
This poll_fn polls the body on the caller. It does not start a task the seed can order against the CSV producer. input.execute has already spawned that producer, and admit_sets charges the endpoint keys later. A wake from Lance spawn_cpu (its own lance-cpu runtime in Lance 13.0.0) or from any other thread changes which of those charges is inside reserved at the refusal.

Inline, crates/omnigraph/src/lib.rs line 19, RIGHT:
graph_manifest_writer_methods_are_not_public_escape_hatches fails on this line. The panic says this pub use is not in SPLIT_CRATE_REEXPORTS. Pin lib.rs: omnigraph_core::dst_blocking next to dst_clock, dst_gate, and dst_ids, or re-export the module pub(crate). Test Workspace and Storage Upgrade Compatibility failed on this commit for this test (run 40477751365).

Inline, crates/omnigraph/src/engine/operators/memory.rs line 361, RIGHT:
QueryWorkLease::own drops the future before the lease. This binding drops the lease at the end of the block, while work is still alive. Use own, as the production poll does. A current-thread runtime does not show this in a replay today.

Inline, crates/omnigraph/src/engine/operators/memory.rs line 1416, RIGHT:
This test checks the thread id. The two calls do not charge one pool, and the test does not compare actual. #[cfg(feature = "dst")] keeps it out of Test Workspace. cargo test -p omnigraph-dst does not run it. One GQT run can pass when both replays happen to match.

@ragnorc

ragnorc commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Closing. #1187 fixed #1160 at the memory-admission design. A csv overwrite now admits its memory once and charges each parse partition against its own share in source order, so a refusal's actual follows that thread's own sequence. That is the owner's fix for this case, and it makes this PR's original purpose moot.

The DST blocking seam is parked on #1161 as a ready implementation, in case a future failing case needs it. The branch stays on the fork at 62285bff. The review's open points are recorded there with it: pin lib.rs: omnigraph_core::dst_blocking in SPLIT_CRATE_REEXPORTS; hold the lease with own in the DST branch; list dst_blocking in the omnigraph-dst crate docs; and narrow the module claim to which thread polls a body, since wakes from Lance's CPU runtime stay outside the seed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: DST replay mismatch, a load's memory refusal reports a timing-dependent actual

1 participant