Repository navigation
fix: recover workflow stores and repair agent runtime behavior - #4773
Merged
ScriptedAlchemy merged 20 commits intoNov 5, 2026
Conversation
A store whose workflow schema this binary refuses reported only that its workflow tables were "absent, incomplete, or not exact", so four refused stores could not be told apart or traced to a cause. The refusal still fails closed through the existing typed reset path but now says what differs: the missing and altered contract tables, the stored workflow identity next to the required one, the differing identity fields, or the first differing column. Only contract names and numbers are echoed; stored SQL, column text, and unrecognized digests never reach the message, and lists are bounded. RefusedAuthorityV1 carries the reason as a Cow so static refusals stay constant. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A backend with no executable bound and a store refusing until its reset cannot recover by repeating work, yet both retried on every tick. Report the unbound executable as a typed backend_unconfigured skip and scheduler status, before any run is admitted, so nothing accumulates and the next tick after a configuration change admits the task. The retry ladder also stops at one attempt for it. Park the loop on a reset refusal until a project open or reconcile wakes it. Log tick and task failures as state changes, which also quiets failures that stay on cadence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The analytics diagnostics summary answered three facts it never measured: an unmounted session-message authority produced `message_count: 0` and four `0.0` per-message rates, an unmounted readiness projection counted every row as skipped and left the fold permanently unavailable, and a window with no observable span served `events_per_hour: 0.0`. The readiness projection was also only mounted by the dashboard MCP handler, so the CLI and MCP admin surfaces always reported it unmounted. It now lives once in `tracedecay-agent-hosts` (where the aggregation already lives), is exported through one idempotent install, and is mounted by the admin-CLI dispatch every diagnostics surface crosses plus the dashboard entry points. `message_count` and every ratio member are now optional on the wire: absent authority and measured zero both serve `null` rather than a fabricated zero, and `events_per_hour` is `null` when no span was observed. Unavailable readiness fallbacks report only genuinely non-`hook_completed` rows as skipped. Dashboard contracts are regenerated and consumers state the absence.
Every tagged release through 1.0.0-beta.92 created the same nine workflow tables and identity digest. beta.93 relaxed the artifact table's CHECK and changed the digest with no path forward, so those stores were refused and could only be reset, losing their workflow records. Admission now recognizes exactly that shape: byte-exact stored SQL for all nine tables, the single published identity row, no other schema object referring to the artifact table, and rows that satisfy the published constraints. It re-proves the shape under the write lock, then moves the table aside, creates the final table, copies rows with their rowids, verifies them byte for byte, drops the old table, compare-and-swaps the digest, and requires the final contract, all in the admission transaction. Anything else keeps the typed refusal, and a store another authority refuses is not touched. The published digest is an opaque label rather than a hash of the stored SQL, so the stored text is the evidence. A failure that recurs on every open is a typed reset refusal; one that can clear, such as a held lock or full disk, stays an operation failure the next open retries. The refusal's identity read gets the same split: a transient read failure stays typed instead of reading as an unreadable identity. The explicit operator request for this upgrade overrides the repository's no-data-migration default for this one published shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two native-runtime diagnostics tests failed in the full application suite while passing alone: `language_refresh` reported `LSP server '/usr/bin/python3' initialize timed out after 5000 ms` and `idle_backfill` never published. Both tests time a 5s initialize budget, and the first interpreter spawn in a test process costs far more than every later one. Measured in this target, the first start against the fixture took 0.49-1.66s while the second and third took 27-90ms for the same script. Under load that one-time cost consumed the whole budget, so the assertion timed the process's first spawn rather than the refresh. Warm the interpreter once before the authority is built, so the measured refresh starts from the steady state. Verified against the configuration that reproduced both failures (24 CPU burners on 18 cores): the full suite now passes 567/567, and the two tests pass 5/5 under 40 burners. The idle backfill also discarded its refresh error entirely, so a backfill that produced nothing looked like one with no work to do. Log it; the broker already records it as the language's engine error.
The reset census rehearsed the whole published-shape upgrade and rolled it back, so a 276 MB store wrote 267 MB to the WAL per census. A census now decides from read-only classification plus the one catalog-only rename no query can predict: 19 ms and no WAL growth at any payload size. One copy statement and one byte-compare query could outrun the transaction deadline, which renews only between statements, so a large store could fail every open and never upgrade, and shutdown could not stop the copy. Rows now move in batches of at most 128 MiB, and the classification scan is paged. The byte compare is gone: SQLite's transaction already guarantees the copy, so rows are checked by count. Classification also missed shapes the copy would have mishandled: a trigger named after the artifact table, triggers on workflow_schema, rows of the wrong storage type, and an identity row that does not decode. A schema that refuses the rename now refuses workflow features and serves the store's other authorities, as other shape refusals do. The upgrade runs last, so a store refused by observations keeps its published shape. A real attach still copies once, since the exact-SQL guard denies the catalog edit that would avoid it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`records_truncated` was `records.len() >= AUTOMATION_RECORD_LIMIT`, which reports the newest page as a truncated window whenever the ledger holds exactly that many rows. A project whose automation ledger held exactly 200 runs was told its counts were partial when the page was the whole ledger. The page read already answers this: `AutomationRunLedgerPageV1::is_complete` is false only when a row was left behind or a row could not be read. Read the page instead of the row list so the flag comes from that authority rather than from an inference about the row count. The new test derives the page bound from the service, then asserts the boundary from both sides: a ledger of exactly that many rows is complete, and one more row is truncated. Against the old predicate it fails with `left: Bool(true), right: false`.
The recent-event tape reported `outcome: "error"` and nothing else, so an operator could see that a call failed but not why. The writer already records a bounded, sanitized `failure_reason` in the event's metadata, and no surface read it back. Serve it on the tape through the same authority the writer used. The text crosses `sanitize_provider_metadata_text`, so text the sanitizer quarantines is omitted rather than echoed, and it is capped at the writer's own 160-char bound so the ceiling is one policy rather than two. Only `error` outcomes serve a reason: a success that happens to carry one in metadata is not a failure to report, and an absent reason stays off the wire. Generated dashboard contracts and the TypeScript SDK still describe the old shape; the scheduler lane owns that regeneration.
The reflector and skill-writer tests that assert `no_session_evidence` inject `EmptyAutomationSessionRetrieval`, whose `retrieve` returns `CompleteZero` without consulting a store. They prove the runner's branch for that outcome, but not that a real, empty production store produces it, so the claim that an empty store reports genuinely-empty evidence rested on a stub. Drive the real production retrieval and the real evidence builder against a freshly registered project store that holds no sessions. The store answers `CompleteZero`, the evidence builder serializes empty hits, and the outcome is a `no_session_evidence` skip carrying a hash, so the skip is attributable rather than an unexplained absence.
The published-shape upgrade ran inside attach, so a large store held daemon admission for the whole copy. This corrects fd310d6 and 8a0dbd0, which still described admission as doing that copy. Daemon attach now only decides: classification plus the catalog-only rename rehearsal, rolled back. The copy runs first in the store's existing schema convergence task, in one transaction, so atomicity, the proof under the write lock, cancellation between statements, and restart from the published shape are unchanged. Workflow storage answers the retryable typed workflow_authority_recovering until that commit; leases issued earlier see the outcome. A store that refuses in the background is held in its reset state and recorded with the other reset-required stores. Short-lived attaches still run the upgrade synchronously. After the commit the dropped table's pages are reclaimed in bounded compactions. Only workflow storage is gated, not the whole session store: daemon boot reads its worker plan from the profile session store, and project opens would be stranded on the degraded core route with no way back to the full server. Measured on a 256 MiB payload, 272 MB store, physical daemon: daemon admission 26 ms; first code hit 941 ms, before the commit at 1886 ms; recover span 1.09 s, of which copy 788 ms, drop 8.75 ms, commit 284 ms; reclaim 1.09 s. WAL peaked at 271 MB and db at 317 MB, settling to 33.5 MB and 272 MB. The store's other writers queue behind the writer lane for the recover span. Also corrects a refusal expectation left stale by 29a6f0c. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A parked scheduler whose run ledger refuses a retired row still read `configured` in the dashboard and `Ready` in `daemon status`, so the operator saw a healthy scheduler and a healthy daemon while every tick was blocked on a store that needs a reset. The scheduler status now serves a closed `blocked_reset_required` availability with the exact reset census: store, authority, reason and the command that resets it. Task rows are withheld while the ledger refuses, because a refused ledger has no authoritative schedule or last-run reading, and the reading recovers as soon as the store does. `daemon status` prints the same census under the protocol line, reports `blocked: reset required`, and `is_ready` no longer claims readiness for a process holding stores reset-required; `is_serving` keeps the narrower "this build answered initialize" reading. A user job bound to the Codex backend with no executable is now skipped as `backend_unconfigured` before external-effect admission instead of recording a failed attempt and a cooldown, so a job runs on the first tick after an executable is configured. A failure stamped with a replaced backend identity no longer imposes cooldown on the current one. Scheduler activation now waits for the project's mandatory runtime publication and re-reads the registered owner, instead of spawning a loop against the core route before the full server exists, and the quiet debounce observes retirement so a parked loop still exits promptly.
The blocked reading was only covered at the ledger helper. This drives the real route through the page: a `blocked_reset_required` status with one refused store must render "reset required", the store's authority, reason and exact remedy, and must never render "configured".
A held lock or full disk rolled the background upgrade back and left workflow storage recovering until the daemon restarted: schedule() runs a store's task once, and a mounted store is not reopened, so 1dba4d9's "next open retries" did not hold for a store that stayed mounted. The store's one convergence task now tries again while its authority is still recovering, about once a minute. It keeps the foreground-open gate, the single convergence permit, and abort on retire and shutdown, and between attempts it holds no permit and only the weak lease issuer, so a waiting store cannot block its owner's retirement. No periodic owner fits: the maintenance tick is hours apart and does not run at all with retention disabled. A store mounted again over a failed upgrade replaces the old worker, which could only find its owner gone. A refusal stays the typed reset; shutdown stops the retries. workflow_storage_problem no longer erases recovering to workflow.unavailable: workflow tools, work retry rebinding, and fan-out recovery answer the retryable workflow.authority_recovering. No wire shape changes. A production-composition journey proves it over a real SQLite write lock and the MCP server: code search serves and workflow tools answer the typed problem while the attempt is parked and after the lock fails it, then the attempt after the lock clears commits on the same mount with every row intact. With the retry disabled this journey fails. SQLITE_FULL is proven as an injected fault at every upgrade step, not a real full disk: the SQLite guard refuses max_page_count, and it stays in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A unix test in the SQLite attachment called a Linux-only helper, so that crate's unit tests did not compile on macOS. The check is now Linux-only, matching the helper. The rest are lint fixes with no behavior change: the daemon reset report builds its section without format-push, and two tests match the outcome directly and show the value when they fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Doctor called a store waiting on its next workflow upgrade attempt stopped, and the only difference between it and a real failure was text in a message. Schema convergence now has a typed retrying state that keeps the last failure, separate from degraded, which now means nothing retries on this mount. Status, the dashboard, and the CLI show it. CLI doctor warns rather than fails, since the daemon clears it by itself. A refusal stays terminal, and the retry delay is unchanged. Three paths could leave a store recovering with no worker, or report retrying with none running: - A retirement refused by a client lease had already stopped the worker, so the restored mount answered recovering until restart. The interrupted plan now runs again on the restored owner. - A retry that woke during a retirement fence exited, though the fence could still roll back. It now waits for the next attempt. - A worker that found its owner gone left retrying on display, and so did a missing timer. Both now report degraded, without overwriting a successor. A retry also takes its strong lease only after the foreground gate and the convergence permit, so queueing no longer holds a client that blocks the store's retirement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ScriptedAlchemy
changed the base branch from
master
to
integration/application-repairs-20261105
November 5, 2026 20:21
ScriptedAlchemy
marked this pull request as ready for review
November 5, 2026 20:22
ScriptedAlchemy
merged commit Nov 5, 2026
45d50cc
into
integration/application-repairs-20261105
25 of 26 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local usage exposed session-store refusals, repeated retries, misleading readiness/analytics, and repository-hint false positives. This change repairs those production paths and preserves operator data.
Validation includes full global-db and application unit suites; focused production recovery, scheduler, Work admission, transport, CLI, and MCP journeys; dashboard tests/typecheck; regenerated contracts and contract checks. Work execution: 28 passed. Latest reported dashboard suite: 2241 passed. Checks were targeted and run at the relevant change points; no full workspace test run is claimed.
Live operator stores were not migrated and the daemon was not replaced. Full-disk rollback uses injected SQLITE_FULL; retry behavior uses a real SQLite lock. The background copy temporarily needs WAL space and holds the store writer during its transaction. Historical active-session reflector emptiness remains unverified; the empty-store test proves only valid CompleteZero behavior.
This work is absent from integration PR #4772 (checked against head 11f4f62). Kept as a separate draft for review and CI.