fix(spacedock): walk the transcript for boot output instead of one head - #224
Draft
iamcxa wants to merge 2 commits into
Draft
fix(spacedock): walk the transcript for boot output instead of one head#224iamcxa wants to merge 2 commits into
iamcxa wants to merge 2 commits into
Conversation
transcript_boot read only the first spacedock_boot_scan_bytes on the stated assumption that boot output is written at session start. That holds for a first officer launched as the agent and not for a session that adopts the role mid-conversation, where boot lands wherever the conversation had reached. On a real transcript the envelope sat 3.3 MB into a 7.0 MB file, so no stage strip ever appeared. Each pass now reads about one budget of whole lines from a per-path cursor and keeps what it found, making the cost once per byte rather than once per byte per refresh. Whole lines through readline, because a record split across passes parses as neither and a tool result carrying an envelope can outgrow the budget on its own. A trailing line without its newline is left for the next pass, since transcripts are read while a harness writes them. Closes #221 Signed-off-by: Kent Chen <kentchen@reccehq.com>
iamcxa
force-pushed
the
fix/spacedock-boot-incremental-scan
branch
from
August 27, 2026 07:40
79862d8 to
9e042f1
Compare
Contributor
CoverageThreshold: |
…ector The cursor tests pin transcript_boot on its own. The user-visible claim is a wiring one -- a declared first officer whose boot output sits past the first pass ends up with a stage strip -- so it needs the collector, the workflow read and the entity read exercised together. Reverting the reader to main's head-only read fails nine tests, two of them these. Signed-off-by: Kent Chen <kentchen@reccehq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #221
What was wrong
transcript_boot()read only the firstspacedock_boot_scan_bytes(512,000) of a transcript, on the assumption its own docstring stated: boot output "is written once at session start and never rewritten".That holds for a first officer launched as the agent. It does not hold for a session that adopts the role mid-conversation by loading the skill — boot then runs wherever the conversation had reached. On the transcript this was found from, the envelope sat 3,306,740 bytes into a 6,995,235-byte file, 6.5× past the window, so no strip ever appeared and the project panel read
This project declares no workflow.What changed
The cursor advances instead of the window widening. Each pass reads about one budget's worth of whole lines from a per-path cursor (
state.spacedock_boot_scan, advanced underscanner_lockliketurns.scan_turns) and keeps what it found. Total cost becomes once per byte instead of once per byte per refresh, and the cliff is gone rather than moved further out.Two details are load-bearing rather than incidental, and each has a test that fails without it:
readline, not a fixed slice. A record split across two passes parses as neither, and a tool result carrying a boot envelope can outgrow the budget on its own — a fixed slice finds no newline in it and stalls the cursor on that line forever.Reaching
spacedock_max_boot_recordspins the cursor, and a transcript walked to its end costs onestat— that is what keeps S-5's per-session Pi cost where it was.Correction to the issue as filed
#221's "Observed in the wild" section overstates what this PR delivers, and I want that on the record rather than discovered in review. It presents a real session as a field case of the strip failure. The transcript facts in it are accurate and this PR does fix them at the function level. But that session carries no
agentSetting, andcollectors/claude.pyreturns atsetting != SPACEDOCK_FObeforetranscript_bootis ever called — so that particular board still shows no strip after this change.The defect #221 names is real and is fixed here. What is not fixed is the launch shape that surfaced it. Making a mid-session Claude FO classifiable at all is a separate decision: it would extend S-5's boot-envelope-as-classifier from Pi to Claude, and S-5's whole point is that an undeclared Claude session opens nothing. That belongs in its own PR with its own argument, not smuggled in here.
Evidence
Deterministic repro (in
SpacedockBootCursorTest): two transcripts carrying the same valid envelope, differing only in where it sits. Before, head-only found0and whole-file found1; after, both find it.The wiring, not just the parser (in
SpacedockDeepBootCollectorTest): a declared first officer with a real workflow README, a mid-flight entity state file, and boot output behind four passes of filler, driven throughcollectors/claude.session_spacedock. The first call returns the role with no workflows; a later one returns the stage strip with its stages and the entity onreview. A shallow-boot case asserts the launch shape that already worked still resolves on the first call. Reverting the reader tomain's head-only read fails nine tests, two of them these.The real transcript, through the new reader: envelope found on pass 7,
definition_dirresolved, cursor at 3,617,790 bytes.Mutation-checked, per CONTRIBUTING. Each mutation applied to
main's behaviour or a boundary, then the suite re-run:>=→>>=→>The first cap test I wrote passed under the "no cap" mutation —
boot_recordscaps each pass on its own, so a transcript small enough to fit one pass could not distinguish the two. It was rewritten to span passes (one envelope in pass one, three in pass two, cap of three) and now fails as it should. Two mutations that survive and are left surviving: the pass budget's<→<=, a one-byte difference in where a pass stops that nothing observable depends on.Gate, run after the rebase onto 2ddf897, with the pinned toolchain from
requirements-dev.txt(ruff 0.16.0, mypy 2.3.0, coverage 7.15.2). The local toolchain on hand was older (ruff 0.15.0, mypy 1.18.2) and reported twoS310findings inquota.pythat the pinned ruff does not — worth knowing before trusting an unpinned local run:Three runs failed before this, always the same test, so I chased it instead of calling it a flake — and the first two explanations I formed were wrong. It is
test_http_api ... test_api_data_names_the_revision_it_served, failing onTimeoutErrorin_get_headers, a 5-second HTTP timeout rather than an assertion.STORE_OVERRIDESis empty by default, so that test's/api/dataruns a real collection over the developer's own stores — about 1.1 s cold on this machine — behind that 5-second timeout. It is reproducible by module pair, and it reproduces identically onmain:So it is pre-existing and local, and
main's one clean full-suite pass was ordering luck rather than immunity. It does not show up in CI because a runner has no harness stores to collect. Worth someone's attention on its own, but not this PR's to fix.I also checked the failure was not my own added cost, since the reader now does per-refresh work
maindid not:maincollect(), real storesIdentical within noise. An earlier reading of 362 ms against 303 ms looked like a regression and was noise from too few samples; I am recording it because I nearly wrote it into this section as one.
Cost
Measured on the 6,995,235-byte transcript above, median of 7:
main)stat, as beforeSo the change buys a one-time 9 ms per large transcript, amortised across refreshes, and then costs nothing.
These numbers come from timing
transcript_bootdirectly, not fromscripts/bench_collect.py, which the pre-PR suite points at and which does not run onmain:AttributeError: 'Namespace' object has no attribute 'host'atcli.py:200, reproduced on a cleanorigin/mainworktree at d3f9867. That looks like--hostlanding without the bench's Namespace being updated. It is unrelated to this diff and not fixed here.Docs
SKILL.md— the strip caveat said no strip appears when boot is "outside the scanned head of a long transcript", which is no longer a thing that happens. It now says the strip can take a few refreshes to arrive on a long transcript.docs/design-spacedock.md— new S-6 for the cursor and the two load-bearing details; S-5's cost paragraph updated; and the rejected-alternatives entry corrected. That entry claimed whole-transcript scanning "would have made the collection pass quadratic in transcript size" — true of the brace-balancing parserboot_recordsreplaced, not of either reader today. The other half of that rejection stands untouched: there are still no laterstatusenvelopes to find, which is a different question from reaching thebootenvelope at all.without-it unanswered
test_an_unreadable_transcript_yields_nothing— the missing-file path was already covered indirectly. It is three lines and pins the defensive-parsing contract explicitly, but nothing has bitten here. Yours to cut.Rebase note
Rebased onto 2ddf897 (#223), which rewrote the
Session titlebullet and addedThe line beneath the title. The conflict was inSKILL.md: #223's three bullets were taken whole and only the strip caveat re-applied on top, so nothing of #223's wording is reverted here.Not done
sync-docswas not invoked; the docs pass above was done by hand, so it has had no voice-and-tone check.