fix(workflow): a start that did not happen must not answer 200 (stacked on #71) - #73
Conversation
fetchWorkflowState answered every IPC failure with
`{ status: 'error', error: error.message }`. That is wrong twice.
'error' is a REAL workflow status. ProcessWorker sets it on a workflow
whose engine failed, and ProcessManager reads it back out of the database.
So "I could not reach the workflow process" and "this workflow failed"
arrived as the same value, and no caller could tell them apart. Both
restart checks in WorkflowService test
`['running','listening','queued'].includes(status)`, so an infrastructure
failure quietly took the not-active branch and the restart was skipped
without anyone being told why.
It also discarded the child's own diagnosis. sendMessage rejects for two
different reasons: the process was unreachable (never spawned, not ready,
failed to initialise, no answer in time), or the child ANSWERED
`{ success: false, error }`. That second message names the actual fault —
`Workflow wf-1 cannot be executed: node "n1" is missing "text"` — and it
was replaced by the single word `error` before any caller saw it. That is
why such failures could not be diagnosed from the API and had to be dug
out of the workflow-process log.
The handler's own catch had been unreachable the whole time. It tests
`error.message.includes('not ready')` and was added on 2026-03-10, two
months after the swallow landed in c9923d2 (2026-01-20): handling written
for an error that structurally could not arrive.
- sendMessage now rejects transport failures with a typed
WorkflowProcessUnavailableError carrying a reason of not-spawned,
not-ready, init-failed or timeout. A failure the child reported stays a
plain Error with the child's message intact.
- fetchWorkflowState logs and rethrows, matching restartActiveWorkflows
in the same class. All three callers already sit inside a try/catch, so
control flow is unchanged where it was already correct.
- GET /workflows/:id/status keeps the 200 'initializing' answer the
original author intended for a process that is still starting — that
branch is now actually reachable — answers 503 'unavailable' with the
reason when the process cannot be reached, and forwards the child's
diagnosis in `details` instead of a generic string.
- Both restart-check catches now report the real error instead of
asserting "not ready" for every cause, including a failure of the
restart itself.
22 tests. Without this change 11 of the 13 bridge tests fail and the 9
route tests cannot load at all, since the typed error does not exist.
…available Self-review: the 'init-failed' path threw away the very thing this change exists to preserve. sendMessage caught the readyPromise rejection and raised a fresh WorkflowProcessUnavailableError without attaching it, so "Workflow process failed to initialize" replaced whatever actually went wrong during spawn — the same erasure, one level down. The error now carries `cause`. The other three reasons are synthesised from a state check and have no underlying error, so `cause` stays absent rather than being set to undefined. 23 tests, +1.
activateWorkflow and deactivateWorkflow answered every IPC failure with
`return { error: error.message }`. The route handlers pass whatever comes
back to `res.json(result)`, and res.json with no status is 200 — so a
start that never happened reported SUCCESS with an error-shaped body.
Every caller gates on the status code, not the body:
WorkflowEngine.vue:268 if (!response.ok) throw ...
Workflows.vue:910,933 if (!response.ok) throw ...
agnt.js SDK axios, which rejects on non-2xx
`response.ok` was true, so none of those failure branches ran. The UI
reported the workflow started. Nothing was armed, no trigger was
listening, and the only trace was a console.error in the backend log.
This is the same defect fixed for fetchWorkflowState, in a worse place:
that one corrupted a status read, these corrupt the Start and Stop
buttons — the two actions where "did it work?" is the entire question.
- Both methods log and rethrow, matching fetchWorkflowState and
restartActiveWorkflows in the same class. All three IPC methods now
report an unreachable process the same way.
- Both route handlers answer 503 with the reason when the process
cannot be reached, and 500 carrying the child's diagnosis when the
process answered with a failure.
- deleteWorkflow isolates its deactivate call. It ran before the row
delete, so letting the new throw escape would have made a workflow
undeletable while the workflow process was down. The row goes either
way, exactly as before; the reason is now logged rather than
discarded.
This also resurrects dead code. The retry around the post-save
reactivate — `catch (reactivateError)`, and the nested `catch
(restoreError)` inside it — could never fire against a method that never
threw, exactly like the `not ready` branch found in the previous change.
Both are live now, and covered.
26 tests. 12 fail without this change. The integration suite deliberately
leaves the bridge REAL, because the defect only existed where the two
halves met: a route test with a mocked bridge cannot see it, and a bridge
test has no response to inspect. It reproduces the original directly —
"expected 200 not to be 200".
There was a problem hiding this comment.
🟡 Changes recommended
activateWorkflow can still resolve { error: ... } from the child (e.g., “already queued/running”), which will again flow to a 200 response unless treated as a thrown failure.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Aligns workflow Start/Stop (and Status) API behavior with actual IPC outcomes by distinguishing an unreachable workflow process from child-reported failures, ensuring callers don’t treat failures as successes due to always-200 JSON responses.
Changes:
- Make
WorkflowProcessBridge.activateWorkflow/deactivateWorkflow/fetchWorkflowStatelog and rethrow IPC failures (instead of returning error-shaped objects). - Map “workflow process unreachable” to 503 (with
reason) and child-reported failures to 500 inWorkflowServicestart/stop/status handlers; keep “not-ready” as 200 initializing for/status. - Add targeted unit + integration tests to reproduce the prior “200 with error-shaped body” regression and pin the new contracts.
File summaries
| File | Description |
|---|---|
| backend/src/workflow/WorkflowProcessBridge.js | Introduces typed “process unavailable” error + predicate; switches IPC helpers to log-and-rethrow for start/stop/status. |
| backend/src/services/WorkflowService.js | Updates route handlers to return 503/500 appropriately for start/stop/status and improves logging; isolates delete from stop failures. |
| backend/src/workflow/WorkflowProcessBridge.unavailable.test.js | Tests typed transport failures vs child-reported failures for sendMessage/fetchWorkflowState. |
| backend/src/workflow/WorkflowProcessBridge.startStop.test.js | Tests start/stop throw on unreachable process and preserve child diagnosis. |
| backend/src/services/WorkflowService.fetchWorkflowState.test.js | Tests /status handler’s 200 initializing vs 503 unavailable vs 500 diagnosis behavior. |
| backend/src/services/WorkflowService.startStop.test.js | Tests /start and /stop handler status mapping and that delete remains successful when stop can’t be confirmed. |
| backend/src/services/WorkflowService.startStopIntegration.test.js | End-to-end reproduction test ensuring start/stop don’t answer 200 when the bridge can’t reach the process. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -268,10 +342,20 @@ class WorkflowProcessBridge { | |||
| return result; | |||
| } catch (error) { | |||
There was a problem hiding this comment.
Good catch, and it was a real hole in the contract this PR's title states — thank you. Verified before changing anything, and there are two such paths, not one:
// ProcessManager.activateWorkflow
return { error: 'Workflow is already queued or running', workflowId }; // early return
return { error: 'Failed to enqueue workflow', workflowId }; // catch blockBoth resolve, so neither reaches the catch block I fixed, and res.json(result) sent both as 200. Making the method throw covered the transport failures and missed these entirely.
Fixed in aaa83606. One deviation from your suggestion, deliberately: I did not throw on a resolved { error }, because the two cases are not the same kind of event. "Already queued or running" is a refusal in the current state with nothing broken; "failed to enqueue" is a real failure. Collapsing both into a 500 would report a healthy, already-running workflow as a server error.
So the handler inspects the payload and maps it — but on a code, not the message text, since matching on English is the coupling the previous commit removed and not worth reintroducing one layer up:
| result | status |
|---|---|
code: 'ALREADY_ACTIVE' |
409 |
| any other code | 500 |
| no code (older shape) | 500 |
ProcessManager now stamps those codes. deactivateWorkflow has no equivalent path — it only ever returns { message, isActive } — verified rather than assumed, so no guard was added there.
Six tests cover it, all failing without the guard. Mutation testing on the new code also caught a weakness in my own test: the "discriminates on the code, not the text" case fed the handler "this workflow is already up and running", which still contained the keyword — so a text-matching mutant passed it and the test proved nothing. Reworded to share no keyword, plus the inverse case, in 60b91ff2. 11 mutants, 11 killed.
Worth adding: the underlying design is the real problem — ProcessManager signalling failure through a resolved payload is what made this invisible in the first place. Converting those to thrown errors would be the proper fix, but it changes the IPC contract and belongs in its own PR.
…view)
Copilot caught the half of this I missed, and it was in the contract this
PR's own title states.
Making activateWorkflow throw covers the transport failures. But
ProcessManager refuses in two places by RESOLVING with an error-shaped
payload instead of throwing:
{ error: 'Workflow is already queued or running', workflowId }
{ error: 'Failed to enqueue workflow', workflowId }
Neither reaches the catch block. `res.json(result)` sent both as 200 —
exactly the lie this change was meant to remove, arriving by the one route
the fix did not cover.
The handler now inspects the resolved payload. Rather than match on the
message text — the coupling removed in the previous commit, and not worth
reintroducing one layer up — ProcessManager stamps a `code`, and the
handler maps it:
ALREADY_ACTIVE -> 409 the request could not be carried out in the
current state, but nothing is broken
anything else -> 500 a real failure
no code -> 500 conservative default for any older shape
deactivateWorkflow has no equivalent path; ProcessManager only ever
returns { message, isActive } from it. Verified rather than assumed.
31 tests, +5: the two refusals, a reworded-message test proving the status
comes from the code, the no-code fallback, and a success case proving a
normal result is not mistaken for a refusal. All 5 fail without the guard.
…ed nothing
Mutation testing: a mutant that discriminated on `result.error.includes('already')`
instead of `result.code` survived. The test meant to forbid exactly that fed it
"this workflow is already up and running, friend" — which still contains the
keyword, so the text-matching mutant read it as the already-running case and
answered 409 anyway.
The wording now shares no keyword with the original, and a second test covers
the inverse: a message that reads like the already-running case must not be
treated as one when the code disagrees.
32 tests, +1. M11 is killed after this.
|
The red Backend check is a pre-existing flake, and I have found its root cause. CI reports It is a millisecond-resolution timestamp race, not an ordering problem: // UpdateScheduler.js:161
const summary = { checkedAt: new Date().toISOString(), ... };// the test, two ticks back to back with nothing between them
await scheduler.tick(); const first = await scheduler.getStatus();
await scheduler.tick(); const second = await scheduler.getStatus();
expect(second.checkedAt).not.toBe(first.checkedAt); // ← fails when both land in the same ms
That matches every observation: it fails intermittently, passes in isolation, and flipped between two consecutive full-suite runs of the same code while I was preparing this PR (295 passed, then 294 passed / 1 failed). It also failed on untouched Deliberately not fixed here — it is unrelated to this change and belongs in its own PR. The fix is small: either inject a clock, or assert on an incrementing counter rather than a wall-clock string. Happy to open that separately if useful. Everything else on this run is green, including the full Playwright suite. |
`UpdateScheduler.tick()` stamps its summary with
checkedAt: new Date().toISOString()
which has millisecond resolution. "replaces the previous pass rather than
accumulating" runs two ticks back to back with nothing slow between them
and asserted only that the two timestamps DIFFER:
expect(second.checkedAt).not.toBe(first.checkedAt);
When both ticks start inside the same millisecond the timestamps are
identical and the assertion fails:
AssertionError: expected '2026-08-22T11:43:29.404Z'
not to be '2026-08-22T11:43:29.404Z'
On this machine the unmodified test fails 23 times out of 30 runs. It has
been red intermittently across every PR opened today, and the failure
carries no hint of a clock — it reads like a logic bug in the scheduler,
which is worse than a test that simply fails.
Fakes Date for that one test and stamps the two passes six hours apart.
Only Date is faked, not setTimeout/setInterval, so the scheduler's real
async filesystem work is untouched and there is nothing to advance.
useFakeTimers is already used in four other backend suites.
The assertion is now stronger, not merely stable: with a controlled clock
it can pin both exact values rather than just their inequality, so a
summary that carried the OLD timestamp forward also fails — which the
previous version would have missed.
Restores real timers in afterEach so a failure inside that test cannot
leak a frozen clock into the rest of the file.
Fixed: 15/15 runs green. Full backend suite: 290/290, twice.
|
Fixed — this PR is now fully green. And one correction to my comment above. I said the At a ~77% failure rate, one passing run is a 23% event, not evidence. It fails in isolation too — the millisecond-clock diagnosis was right, the "order-dependent" framing was not. The failure is exactly as predicted: Fixed in #74, standalone against That commit is cherry-picked here as Verification: fixed test 15/15 green; full backend suite 295/295 on this branch, and 290/290 twice on the #74 branch. All 5 checks green here now. |
Prepares the client and the tenant path for the token-proof flip, and writes
down what the /auth surface now actually does.
remoteTokenVerifier — THE BLOCKER
A hosted tenant delegates verification to api.agnt.gg, and every non-2xx from
the issuer was classified "unreachable". Once token-proof enforcement is
turned on, a token without a proof claim starts getting 401 there — which
would have meant:
- the stale-grace window kept serving a REFUSED token for up to 30 more
minutes, on exactly the installs that are reachable from the internet;
- the refusal was counted as remoteFail, so a wave of legitimate denials
would read on the dashboard as "the issuer is down";
- it was never cached as a denial, so a rejected client re-asked on every
request.
401/403 is now a denial: no grace, cached, counted as remoteDeny. 429 and 5xx
still mean UNKNOWN, because a rate-limited issuer must not log everyone out.
Four new cases fail against the previous code.
OAuth callback copy
POST /auth/callback answers a stable `reason`; the client rendered
`errorData.error` verbatim, which is what put a SQLite constraint string in
front of a user. New services/oauthCallbackErrors.js maps each reason to a
sentence that ends in an instruction, shared by the two consumers. A body with
no reason falls back rather than surfacing prose nobody wrote, and a transport
failure is passed through untouched — axios already describes it accurately.
OAuthCallback.spec.js asserted the raw server string appeared on screen, so it
was pinning the defect. It now asserts the opposite, plus that no driver string
can reach the screen and that a non-JSON error body still renders.
NOT DONE, DELIBERATELY: `reason: 'proof'` was not added to the axios
interceptor. That interceptor is scoped to BASE_URL, the local backend verifies
with the shared secret and cannot produce that reason, and every path that can
already resolves correctly — desktop's fetchUserData classifies the remote 401
as http_401 (definitive), and a hosted tenant now returns 'invalid' via the fix
above. Adding it would have been inert, which is the same mistake #76 made.
Documentation
- The two middlewares are distinguished: `authenticateToken` identifies and
continues, `requireAuth` refuses. The doc previously said "Required" for both.
- The `provider:nonce:origin` state format, and why the nonce is in the middle.
- The full callback reason vocabulary, with which statuses are retryable.
- Remote Feedback Routes: live and entirely undocumented until now.
- Stream base path corrected from /streams to /stream — verified against
production, /streams 404s.
- The /start, /stop and /:id/status contract change from #71 and #73, including
that the SDK is axios and will now throw where it used to swallow.
- 4 further undocumented live routes closed. A parse-the-router audit now
reports 35/35 documented and 0 broken TOC anchors.
Backend 4735 tests (1 pre-existing flake in runJournal.heartbeat, filed,
passes 13/13 in isolation). Frontend 3943, all pass.
Important
Stacked on #71 — merge that first.
This branch is cut from #71, so the diff shows its two commits as well. Only
835367f9belongs to this PR. Once #71 lands, this reduces to that single commit; if #71 is revised, I will rebase.It depends on the
WorkflowProcessUnavailableErrorclass and theisWorkflowProcessUnavailablepredicate introduced there. Building it onmainwould have meant inventing both twice.What
activateWorkflowanddeactivateWorkflowanswered every IPC failure withreturn { error: error.message }. The route handlers pass the result straight tores.json(result)— andres.jsonwith no status is 200.So a start that never happened answered success, with an error-shaped body.
Why it mattered
Every caller gates on the status code, not the body:
WorkflowEngine.vue:268if (!response.ok) throwWorkflows.vue:910,933if (!response.ok) throwagnt.jsSDKresponse.okwas true, so none of those failure branches ran. The UI reported the workflow started. Nothing was armed, no trigger was listening, and the only trace was aconsole.errorin the backend log.This is the same defect #71 fixed for
fetchWorkflowState, in a worse place. That one corrupted a status read. These corrupt the Start and Stop buttons — the two actions where "did it work?" is the entire question.How
fetchWorkflowStateandrestartActiveWorkflowsin the same class. All three IPC methods now report an unreachable process the same way — a test asserts exactly that, since the class previously answered the same failure two different ways depending on which method you called.reasonwhen the process cannot be reached, and 500 carrying the child's diagnosis when the process answered with a failure. Identical shape to fix(workflow): an unreachable workflow process is not a workflow state #71's/statushandler.deleteWorkflowisolates its deactivate call. It runs before the row delete, so letting the new throw escape would have made a workflow undeletable while the workflow process was down. The row goes either way, exactly as before — what changes is that the reason is logged rather than discarded.It also resurrects dead code
The retry around the post-save reactivate has never been able to fire:
Same pattern as the
not readybranch found in #71 — handling written for an error that structurally could not arrive. Both are live now, and covered by tests.Worth noting the interaction with #71: when the process is unreachable,
fetchWorkflowStatenow throws first and the outer catch skips the restart check entirely, so these retries are only reached when the process is up and genuinely refused. No added latency against a down process.Testing
26 tests across 3 files. 12 fail without this change.
The third file is the one that matters most. The other two test each half in isolation — the bridge throws, the handler maps a throw to 503/500 — but neither reproduces the actual defect, which only existed where the halves met:
A route test with a mocked bridge cannot see that, and a bridge test has no
resto inspect. SoWorkflowService.startStopIntegration.test.jsmocks the models, database and realtime layer and deliberately leavesWorkflowProcessBridgereal. Against the base commit it fails with:Mutation tested — 8 mutants, 8 killed, 0 survived:
reasonfrom the 503 bodyFull backend suite: 295 passed / 295 files, zero failures. (The
UpdateScheduler.statusflake noted on #70 and #71 did not fire on this run — further evidence it is order-dependent rather than caused by any of these changes.)Notes for the reviewer
POST /workflows/:id/startand/stopcan now answer 503 or 500 where they previously answered 200. That is the point — those 200s were false — but it deserves a deliberate nod. All three known callers already handle non-2xx correctly, so no client change is needed.activateWorkflowis also called from the two restart paths insaveWorkflow/updateWorkflow. Both already sat insidetry/catch, so their control flow is unchanged — only their error messages improve.