Skip to content

fix(workflow): a start that did not happen must not answer 200 (stacked on #71) - #73

Merged
agnt-gg merged 6 commits into
agnt-gg:mainfrom
rimusz:fix/workflow-start-stop-report-failure
Aug 23, 2026
Merged

fix(workflow): a start that did not happen must not answer 200 (stacked on #71)#73
agnt-gg merged 6 commits into
agnt-gg:mainfrom
rimusz:fix/workflow-start-stop-report-failure

Conversation

@rimusz

@rimusz rimusz commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Important

Stacked on #71 — merge that first.

This branch is cut from #71, so the diff shows its two commits as well. Only 835367f9 belongs to this PR. Once #71 lands, this reduces to that single commit; if #71 is revised, I will rebase.

It depends on the WorkflowProcessUnavailableError class and the isWorkflowProcessUnavailable predicate introduced there. Building it on main would have meant inventing both twice.

What

activateWorkflow and deactivateWorkflow answered every IPC failure with return { error: error.message }. The route handlers pass the result straight to res.json(result) — and res.json with no status is 200.

So a start that never happened answered success, with an error-shaped body.

Why it mattered

Every caller gates on the status code, not the body:

caller check
WorkflowEngine.vue:268 if (!response.ok) throw
Workflows.vue:910,933 if (!response.ok) throw
agnt.js SDK axios — rejects on non-2xx

response.ok was true, so none of those failure branches ran. The UI reported the workflow started. Nothing was armed, no trigger was listening, and the only trace was a console.error in the backend log.

This is the same defect #71 fixed for fetchWorkflowState, in a worse place. That one corrupted a status read. These corrupt the Start and Stop buttons — the two actions where "did it work?" is the entire question.

How

  • Both bridge methods log and rethrow, matching fetchWorkflowState and restartActiveWorkflows in the same class. All three IPC methods now report an unreachable process the same way — a test asserts exactly that, since the class previously answered the same failure two different ways depending on which method you called.
  • Both route handlers answer 503 with the reason when the process cannot be reached, and 500 carrying the child's diagnosis when the process answered with a failure. Identical shape to fix(workflow): an unreachable workflow process is not a workflow state #71's /status handler.
  • deleteWorkflow isolates its deactivate call. It runs before the row delete, so letting the new throw escape would have made a workflow undeletable while the workflow process was down. The row goes either way, exactly as before — what changes is that the reason is logged rather than discarded.

It also resurrects dead code

The retry around the post-save reactivate has never been able to fire:

try {
  await WorkflowProcessBridge.activateWorkflow(updatedWorkflow, userId);
} catch (reactivateError) {                    // ← unreachable: activate never threw
  console.error('Failed to reactivate workflow after save:', reactivateError.message);
  try {
    await WorkflowProcessBridge.activateWorkflow(updatedWorkflow, userId);
  } catch (restoreError) {                     // ← also unreachable
    console.error('Could not restore workflow active state:', restoreError.message);
  }
}

Same pattern as the not ready branch found in #71 — handling written for an error that structurally could not arrive. Both are live now, and covered by tests.

Worth noting the interaction with #71: when the process is unreachable, fetchWorkflowState now throws first and the outer catch skips the restart check entirely, so these retries are only reached when the process is up and genuinely refused. No added latency against a down process.

Testing

26 tests across 3 files. 12 fail without this change.

The third file is the one that matters most. The other two test each half in isolation — the bridge throws, the handler maps a throw to 503/500 — but neither reproduces the actual defect, which only existed where the halves met:

bridge swallows -> returns { error: message } -> res.json(result) -> 200

A route test with a mocked bridge cannot see that, and a bridge test has no res to inspect. So WorkflowService.startStopIntegration.test.js mocks the models, database and realtime layer and deliberately leaves WorkflowProcessBridge real. Against the base commit it fails with:

AssertionError: expected 200 not to be 200
AssertionError: expected true to be false      ← error-shaped body with a success status

Mutation tested — 8 mutants, 8 killed, 0 survived:

mutant result
restore the activate swallow killed (6)
restore the deactivate swallow killed (3)
drop the 503 branch on start killed (3)
drop the 503 branch on stop killed (2)
let the delete deactivate failure escape killed (3)
stop logging why the delete could not confirm killed (1)
stop forwarding the diagnosis on a start failure killed (1)
drop reason from the 503 body killed (2)

Full backend suite: 295 passed / 295 files, zero failures. (The UpdateScheduler.status flake noted on #70 and #71 did not fire on this run — further evidence it is order-dependent rather than caused by any of these changes.)

Notes for the reviewer

  • This is a behaviour change at the API boundary. POST /workflows/:id/start and /stop can now answer 503 or 500 where they previously answered 200. That is the point — those 200s were false — but it deserves a deliberate nod. All three known callers already handle non-2xx correctly, so no client change is needed.
  • A product question I deliberately did not decide: should deleting a workflow fail when its triggers cannot be confirmed stopped? Today it proceeds, and I preserved that. The argument for failing is orphaned live triggers; the argument against is a workflow you cannot remove while the process is down. Worth a separate discussion rather than a silent change here.
  • activateWorkflow is also called from the two restart paths in saveWorkflow/updateWorkflow. Both already sat inside try/catch, so their control flow is unchanged — only their error messages improve.

rimusz added 3 commits August 22, 2026 12:53
fetchWorkflowState answered every IPC failure with
`{ status: 'error', error: error.message }`. That is wrong twice.

'error' is a REAL workflow status. ProcessWorker sets it on a workflow
whose engine failed, and ProcessManager reads it back out of the database.
So "I could not reach the workflow process" and "this workflow failed"
arrived as the same value, and no caller could tell them apart. Both
restart checks in WorkflowService test
`['running','listening','queued'].includes(status)`, so an infrastructure
failure quietly took the not-active branch and the restart was skipped
without anyone being told why.

It also discarded the child's own diagnosis. sendMessage rejects for two
different reasons: the process was unreachable (never spawned, not ready,
failed to initialise, no answer in time), or the child ANSWERED
`{ success: false, error }`. That second message names the actual fault —
`Workflow wf-1 cannot be executed: node "n1" is missing "text"` — and it
was replaced by the single word `error` before any caller saw it. That is
why such failures could not be diagnosed from the API and had to be dug
out of the workflow-process log.

The handler's own catch had been unreachable the whole time. It tests
`error.message.includes('not ready')` and was added on 2026-03-10, two
months after the swallow landed in c9923d2 (2026-01-20): handling written
for an error that structurally could not arrive.

  - sendMessage now rejects transport failures with a typed
    WorkflowProcessUnavailableError carrying a reason of not-spawned,
    not-ready, init-failed or timeout. A failure the child reported stays a
    plain Error with the child's message intact.
  - fetchWorkflowState logs and rethrows, matching restartActiveWorkflows
    in the same class. All three callers already sit inside a try/catch, so
    control flow is unchanged where it was already correct.
  - GET /workflows/:id/status keeps the 200 'initializing' answer the
    original author intended for a process that is still starting — that
    branch is now actually reachable — answers 503 'unavailable' with the
    reason when the process cannot be reached, and forwards the child's
    diagnosis in `details` instead of a generic string.
  - Both restart-check catches now report the real error instead of
    asserting "not ready" for every cause, including a failure of the
    restart itself.

22 tests. Without this change 11 of the 13 bridge tests fail and the 9
route tests cannot load at all, since the typed error does not exist.
…available

Self-review: the 'init-failed' path threw away the very thing this change
exists to preserve. sendMessage caught the readyPromise rejection and
raised a fresh WorkflowProcessUnavailableError without attaching it, so
"Workflow process failed to initialize" replaced whatever actually went
wrong during spawn — the same erasure, one level down.

The error now carries `cause`. The other three reasons are synthesised from
a state check and have no underlying error, so `cause` stays absent rather
than being set to undefined.

23 tests, +1.
activateWorkflow and deactivateWorkflow answered every IPC failure with
`return { error: error.message }`. The route handlers pass whatever comes
back to `res.json(result)`, and res.json with no status is 200 — so a
start that never happened reported SUCCESS with an error-shaped body.

Every caller gates on the status code, not the body:

    WorkflowEngine.vue:268   if (!response.ok) throw ...
    Workflows.vue:910,933    if (!response.ok) throw ...
    agnt.js SDK              axios, which rejects on non-2xx

`response.ok` was true, so none of those failure branches ran. The UI
reported the workflow started. Nothing was armed, no trigger was
listening, and the only trace was a console.error in the backend log.

This is the same defect fixed for fetchWorkflowState, in a worse place:
that one corrupted a status read, these corrupt the Start and Stop
buttons — the two actions where "did it work?" is the entire question.

  - Both methods log and rethrow, matching fetchWorkflowState and
    restartActiveWorkflows in the same class. All three IPC methods now
    report an unreachable process the same way.
  - Both route handlers answer 503 with the reason when the process
    cannot be reached, and 500 carrying the child's diagnosis when the
    process answered with a failure.
  - deleteWorkflow isolates its deactivate call. It ran before the row
    delete, so letting the new throw escape would have made a workflow
    undeletable while the workflow process was down. The row goes either
    way, exactly as before; the reason is now logged rather than
    discarded.

This also resurrects dead code. The retry around the post-save
reactivate — `catch (reactivateError)`, and the nested `catch
(restoreError)` inside it — could never fire against a method that never
threw, exactly like the `not ready` branch found in the previous change.
Both are live now, and covered.

26 tests. 12 fail without this change. The integration suite deliberately
leaves the bridge REAL, because the defect only existed where the two
halves met: a route test with a mocked bridge cannot see it, and a bridge
test has no response to inspect. It reproduces the original directly —
"expected 200 not to be 200".
Copilot AI lite review requested due to automatic review settings August 22, 2026 11:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

activateWorkflow can still resolve { error: ... } from the child (e.g., “already queued/running”), which will again flow to a 200 response unless treated as a thrown failure.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Aligns workflow Start/Stop (and Status) API behavior with actual IPC outcomes by distinguishing an unreachable workflow process from child-reported failures, ensuring callers don’t treat failures as successes due to always-200 JSON responses.

Changes:

  • Make WorkflowProcessBridge.activateWorkflow / deactivateWorkflow / fetchWorkflowState log and rethrow IPC failures (instead of returning error-shaped objects).
  • Map “workflow process unreachable” to 503 (with reason) and child-reported failures to 500 in WorkflowService start/stop/status handlers; keep “not-ready” as 200 initializing for /status.
  • Add targeted unit + integration tests to reproduce the prior “200 with error-shaped body” regression and pin the new contracts.
File summaries
File Description
backend/src/workflow/WorkflowProcessBridge.js Introduces typed “process unavailable” error + predicate; switches IPC helpers to log-and-rethrow for start/stop/status.
backend/src/services/WorkflowService.js Updates route handlers to return 503/500 appropriately for start/stop/status and improves logging; isolates delete from stop failures.
backend/src/workflow/WorkflowProcessBridge.unavailable.test.js Tests typed transport failures vs child-reported failures for sendMessage/fetchWorkflowState.
backend/src/workflow/WorkflowProcessBridge.startStop.test.js Tests start/stop throw on unreachable process and preserve child diagnosis.
backend/src/services/WorkflowService.fetchWorkflowState.test.js Tests /status handler’s 200 initializing vs 503 unavailable vs 500 diagnosis behavior.
backend/src/services/WorkflowService.startStop.test.js Tests /start and /stop handler status mapping and that delete remains successful when stop can’t be confirmed.
backend/src/services/WorkflowService.startStopIntegration.test.js End-to-end reproduction test ensuring start/stop don’t answer 200 when the bridge can’t reach the process.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 337 to 343
@@ -268,10 +342,20 @@ class WorkflowProcessBridge {
return result;
} catch (error) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and it was a real hole in the contract this PR's title states — thank you. Verified before changing anything, and there are two such paths, not one:

// ProcessManager.activateWorkflow
return { error: 'Workflow is already queued or running', workflowId };  // early return
return { error: 'Failed to enqueue workflow', workflowId };             // catch block

Both resolve, so neither reaches the catch block I fixed, and res.json(result) sent both as 200. Making the method throw covered the transport failures and missed these entirely.

Fixed in aaa83606. One deviation from your suggestion, deliberately: I did not throw on a resolved { error }, because the two cases are not the same kind of event. "Already queued or running" is a refusal in the current state with nothing broken; "failed to enqueue" is a real failure. Collapsing both into a 500 would report a healthy, already-running workflow as a server error.

So the handler inspects the payload and maps it — but on a code, not the message text, since matching on English is the coupling the previous commit removed and not worth reintroducing one layer up:

result status
code: 'ALREADY_ACTIVE' 409
any other code 500
no code (older shape) 500

ProcessManager now stamps those codes. deactivateWorkflow has no equivalent path — it only ever returns { message, isActive } — verified rather than assumed, so no guard was added there.

Six tests cover it, all failing without the guard. Mutation testing on the new code also caught a weakness in my own test: the "discriminates on the code, not the text" case fed the handler "this workflow is already up and running", which still contained the keyword — so a text-matching mutant passed it and the test proved nothing. Reworded to share no keyword, plus the inverse case, in 60b91ff2. 11 mutants, 11 killed.

Worth adding: the underlying design is the real problem — ProcessManager signalling failure through a resolved payload is what made this invisible in the first place. Converting those to thrown errors would be the proper fix, but it changes the IPC contract and belongs in its own PR.

rimusz added 2 commits August 22, 2026 14:31
…view)

Copilot caught the half of this I missed, and it was in the contract this
PR's own title states.

Making activateWorkflow throw covers the transport failures. But
ProcessManager refuses in two places by RESOLVING with an error-shaped
payload instead of throwing:

    { error: 'Workflow is already queued or running', workflowId }
    { error: 'Failed to enqueue workflow', workflowId }

Neither reaches the catch block. `res.json(result)` sent both as 200 —
exactly the lie this change was meant to remove, arriving by the one route
the fix did not cover.

The handler now inspects the resolved payload. Rather than match on the
message text — the coupling removed in the previous commit, and not worth
reintroducing one layer up — ProcessManager stamps a `code`, and the
handler maps it:

    ALREADY_ACTIVE  -> 409  the request could not be carried out in the
                            current state, but nothing is broken
    anything else   -> 500  a real failure
    no code         -> 500  conservative default for any older shape

deactivateWorkflow has no equivalent path; ProcessManager only ever
returns { message, isActive } from it. Verified rather than assumed.

31 tests, +5: the two refusals, a reworded-message test proving the status
comes from the code, the no-code fallback, and a success case proving a
normal result is not mistaken for a refusal. All 5 fail without the guard.
…ed nothing

Mutation testing: a mutant that discriminated on `result.error.includes('already')`
instead of `result.code` survived. The test meant to forbid exactly that fed it
"this workflow is already up and running, friend" — which still contains the
keyword, so the text-matching mutant read it as the already-running case and
answered 409 anyway.

The wording now shares no keyword with the original, and a second test covers
the inverse: a message that reads like the already-running case must not be
treated as one when the code disagrees.

32 tests, +1. M11 is killed after this.
@rimusz

rimusz commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

The red Backend check is a pre-existing flake, and I have found its root cause.

CI reports 1 failed | 294 passed (295). The single failure is backend/src/plugins/UpdateScheduler.status.test.js > replaces the previous pass rather than accumulating, which imports nothing this PR touches.

It is a millisecond-resolution timestamp race, not an ordering problem:

// UpdateScheduler.js:161
const summary = { checkedAt: new Date().toISOString(), ... };
// the test, two ticks back to back with nothing between them
await scheduler.tick();  const first  = await scheduler.getStatus();
await scheduler.tick();  const second = await scheduler.getStatus();
expect(second.checkedAt).not.toBe(first.checkedAt);   // ← fails when both land in the same ms

toISOString() has millisecond resolution, so the assertion only passes when the two ticks happen to straddle a millisecond boundary. On this machine, two back-to-back new Date().toISOString() calls are identical in ~100% of 200k trials — the test passes only when something slow enough intervenes.

That matches every observation: it fails intermittently, passes in isolation, and flipped between two consecutive full-suite runs of the same code while I was preparing this PR (295 passed, then 294 passed / 1 failed). It also failed on untouched origin/main when I checked during #70.

Deliberately not fixed here — it is unrelated to this change and belongs in its own PR. The fix is small: either inject a clock, or assert on an incrementing counter rather than a wall-clock string. Happy to open that separately if useful.

Everything else on this run is green, including the full Playwright suite.

`UpdateScheduler.tick()` stamps its summary with

    checkedAt: new Date().toISOString()

which has millisecond resolution. "replaces the previous pass rather than
accumulating" runs two ticks back to back with nothing slow between them
and asserted only that the two timestamps DIFFER:

    expect(second.checkedAt).not.toBe(first.checkedAt);

When both ticks start inside the same millisecond the timestamps are
identical and the assertion fails:

    AssertionError: expected '2026-08-22T11:43:29.404Z'
                 not to be '2026-08-22T11:43:29.404Z'

On this machine the unmodified test fails 23 times out of 30 runs. It has
been red intermittently across every PR opened today, and the failure
carries no hint of a clock — it reads like a logic bug in the scheduler,
which is worse than a test that simply fails.

Fakes Date for that one test and stamps the two passes six hours apart.
Only Date is faked, not setTimeout/setInterval, so the scheduler's real
async filesystem work is untouched and there is nothing to advance.
useFakeTimers is already used in four other backend suites.

The assertion is now stronger, not merely stable: with a controlled clock
it can pin both exact values rather than just their inequality, so a
summary that carried the OLD timestamp forward also fails — which the
previous version would have missed.

Restores real timers in afterEach so a failure inside that test cannot
leak a frozen clock into the rest of the file.

Fixed: 15/15 runs green. Full backend suite: 290/290, twice.
@rimusz

rimusz commented Aug 22, 2026

Copy link
Copy Markdown
Contributor Author

Fixed — this PR is now fully green. And one correction to my comment above.

I said the UpdateScheduler test "passes in isolation", which pointed at ordering. That was based on a single isolated run. Measuring it properly on pristine main:

unmodified test, 30 isolated runs  ->  23 failed

At a ~77% failure rate, one passing run is a 23% event, not evidence. It fails in isolation too — the millisecond-clock diagnosis was right, the "order-dependent" framing was not.

The failure is exactly as predicted:

AssertionError: expected '2026-08-22T11:43:29.404Z' not to be '2026-08-22T11:43:29.404Z'

Fixed in #74, standalone against main so it can merge on its own and unblock every other PR — it fakes Date for that one test (only Date, so the scheduler's real async filesystem work is untouched) and stamps the two passes six hours apart. The assertion also gets stronger rather than merely stable: it now pins both exact timestamps instead of their inequality, which additionally catches a summary that carried the old timestamp forward — the very case the test exists to detect, and one not.toBe would have passed.

That commit is cherry-picked here as d17fdb04 so this PR is green now rather than waiting on #74. It is the identical patch, so it dedupes when #74 merges.

Verification: fixed test 15/15 green; full backend suite 295/295 on this branch, and 290/290 twice on the #74 branch.

All 5 checks green here now.

@agnt-gg
agnt-gg merged commit dabc9c8 into agnt-gg:main Aug 23, 2026
5 checks passed
agnt-gg added a commit that referenced this pull request Aug 23, 2026
Prepares the client and the tenant path for the token-proof flip, and writes
down what the /auth surface now actually does.

remoteTokenVerifier — THE BLOCKER
A hosted tenant delegates verification to api.agnt.gg, and every non-2xx from
the issuer was classified "unreachable". Once token-proof enforcement is
turned on, a token without a proof claim starts getting 401 there — which
would have meant:

  - the stale-grace window kept serving a REFUSED token for up to 30 more
    minutes, on exactly the installs that are reachable from the internet;
  - the refusal was counted as remoteFail, so a wave of legitimate denials
    would read on the dashboard as "the issuer is down";
  - it was never cached as a denial, so a rejected client re-asked on every
    request.

401/403 is now a denial: no grace, cached, counted as remoteDeny. 429 and 5xx
still mean UNKNOWN, because a rate-limited issuer must not log everyone out.
Four new cases fail against the previous code.

OAuth callback copy
POST /auth/callback answers a stable `reason`; the client rendered
`errorData.error` verbatim, which is what put a SQLite constraint string in
front of a user. New services/oauthCallbackErrors.js maps each reason to a
sentence that ends in an instruction, shared by the two consumers. A body with
no reason falls back rather than surfacing prose nobody wrote, and a transport
failure is passed through untouched — axios already describes it accurately.

OAuthCallback.spec.js asserted the raw server string appeared on screen, so it
was pinning the defect. It now asserts the opposite, plus that no driver string
can reach the screen and that a non-JSON error body still renders.

NOT DONE, DELIBERATELY: `reason: 'proof'` was not added to the axios
interceptor. That interceptor is scoped to BASE_URL, the local backend verifies
with the shared secret and cannot produce that reason, and every path that can
already resolves correctly — desktop's fetchUserData classifies the remote 401
as http_401 (definitive), and a hosted tenant now returns 'invalid' via the fix
above. Adding it would have been inert, which is the same mistake #76 made.

Documentation
- The two middlewares are distinguished: `authenticateToken` identifies and
  continues, `requireAuth` refuses. The doc previously said "Required" for both.
- The `provider:nonce:origin` state format, and why the nonce is in the middle.
- The full callback reason vocabulary, with which statuses are retryable.
- Remote Feedback Routes: live and entirely undocumented until now.
- Stream base path corrected from /streams to /stream — verified against
  production, /streams 404s.
- The /start, /stop and /:id/status contract change from #71 and #73, including
  that the SDK is axios and will now throw where it used to swallow.
- 4 further undocumented live routes closed. A parse-the-router audit now
  reports 35/35 documented and 0 broken TOC anchors.

Backend 4735 tests (1 pre-existing flake in runJournal.heartbeat, filed,
passes 13/13 in isolation). Frontend 3943, all pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants