Bot orchestration v2: state machine, Flue 1.0, free-text classifier, evals - #1606
Closed
ascorbic wants to merge 16 commits into
Closed
Bot orchestration v2: state machine, Flue 1.0, free-text classifier, evals#1606ascorbic wants to merge 16 commits into
ascorbic wants to merge 16 commits into
Conversation
defineWorkflow/defineAgent; investigate collapses three agents onto one workflow agent with separate classify/fix sessions and per-op model overrides.
machine.ts single source of truth, router.cjs pure logic + tests, orchestrate.yml (brain) and investigate-run.yml (executor), read-only linter and drift check. Gated behind BOT_STATE_MACHINE_V2.
The bot's @emdashbot grammar is now free-text by default. parseCommand only fires on an EXACT bare verb (extra words route to the model), so prose containing a verb can't accidentally trigger an action. Arg-carrying intents (implement <directive>, revise <feedback>) flow through one state-aware classifier (classify-command), constrained to the current state's offered commands minus the destructive ones (decline, take_over) -- those still require an exact bare verb, so free text can never silently close or disengage an item. Replaces the two narrow classifiers (classify-reply, classify-maintainer-reply) with one state-aware workflow that returns a machine event directly. orchestrate.yml now calls resolveComment once and dispatches the classifier only on free text. Adds an eval suite (Sentry vitest-evals) with a 43-case labeled dataset shared between the vitest gate and a model-sweep harness. Sweep records pass rate, latency, tokens, and cost (from the live Workers AI price sheet). Switches the classifier default to qwen3-30b-a3b-fp8 (~9x cheaper, ~4.4x faster than kimi-k2.7-code at near-tied accuracy on the 43-case dataset, with no capacity timeouts). FLUE_CLASSIFIER_MODEL still overrides per run. Fix stage stays on kimi-k2.7-code (classifier evals don't speak to code-fix quality). To reach the current Workers AI catalog from Node (pi-ai's cloudflare-ai-gateway catalog is a stale snapshot and cloudflare-workers-ai is binding-only), registers a cf-wai HTTP provider in app.ts against the gateway /compat endpoint. Workers-AI-only by token scope; full current model lineup available. Verified: flue build, tsc, 23 router tests, generate --check, YAML parse.
|
github-actions
Bot
added
review/needs-review
Contributor
Scope checkThis PR changes 5,006 lines across 28 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | 064e79f | Jun 25 2026, 06:17 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | f73cef6 | Jun 25 2026, 02:31 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | f73cef6 | Jun 25 2026, 02:31 PM |
github-actions
Bot
added
review/needs-rereview
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
The V2 bot state-machine is a sensible architectural direction and the router/tests are solid, but this PR has three concrete blockers that make merging as-is unsafe:
-
The new executor workflow is a functional placeholder.
.github/workflows/investigate-run.ymlexplicitly says it elided the runner provisioning steps (pnpm, root deps, Playwright/agent-browser, package builds) “for review clarity.” As committed, the workflow would fail whenBOT_STATE_MACHINE_V2is flipped because the agent sandbox has none of the tooling it needs to reproduce/fix. A gated-but-broken file is not a safe cutover state. -
Backwards-compatibility of the Flue 1.0 migration is unresolved.
.flue/scripts/run-local.tsdocuments that Flue 1.0 renamed--payloadto--input. The legacyinvestigate.yml,reporter-reply.yml, andmaintainer-reply.ymlstill invokeflue run … --payload, and they are not gated byBOT_STATE_MACHINE_V2. Unless the Flue 1.0 beta CLI still accepts--payload(unverified from repo sources), the live bot breaks immediately on merge. Even if--payloadis aliased, enabling V2 will double-fire runs:investigate.ymlandorchestrate.ymlboth react to thebot:reprolabel. The old workflows need either migration to the new CLI contract or a V2-off guard. -
Shell injection via an AI-generated commit message.
investigate-run.ymlrunsgit commit -m "$MSG", expanding the agent-produced commit message in double quotes. A crafted issue/body can produce backticks or$(…)in that message, executing arbitrary commands in the runner.
I also found smaller correctness/style issues: an empty @emdashbot mention gets routed to the classifier and fails validation, and two stale comments misstate the default model / entry state.
I read the diff full-length and traced the router, workflows, and Flue migration statically; I did not run the test suite or Flue CLI.
| fetch-depth: 0 | ||
| ref: ${{ env.MODE == 'revise' && format('bot/fix-{0}', env.ISSUE_NUMBER) || '' }} | ||
|
|
||
| # NOTE: the full runner provisioning (Node, pnpm, repo install + build, |
Contributor
There was a problem hiding this comment.
[needs fixing] The executor workflow is committed with a placeholder comment that says all runner provisioning (pnpm, root dependencies, Playwright/agent-browser, package builds, dev-server setup) is “elided here for review clarity” and will be added at cutover. As committed, only Node, pnpm, and the .flue agent are installed, so the agent sandbox will not be able to run pnpm dev, pnpm test, or agent-browser. A gated-but-broken file is not a safe merge state; include the proven provisioning steps from the existing investigate.yml before this workflow is added to the repo.
|
|
||
| // `pnpm exec` (not `npx`) so we invoke the lockfile-pinned Flue. | ||
| // `flue run` in 0.8 generates the workflow run id itself; no --id flag. | ||
| // Flue 1.0: `flue run <workflow> --input '<json>'` (was `--payload`). |
Contributor
There was a problem hiding this comment.
[needs fixing] This comment documents that Flue 1.0 renamed --payload to --input. The legacy investigate.yml, reporter-reply.yml, and maintainer-reply.yml still invoke flue run … --payload, and they are not gated by BOT_STATE_MACHINE_V2. If the Flue 1.0 beta CLI no longer accepts --payload, the live bot breaks the moment this PR merges. Additionally, once V2 is enabled, investigate.yml and orchestrate.yml will both handle the bot:repro label, producing duplicate agent runs and conflicting label flips. Please either confirm --payload is still aliased and the default-export defineWorkflow contract is backward-compatible with the old YAML, or migrate the legacy workflows to --input and add a vars.BOT_STATE_MACHINE_V2 != '1' guard to them.
| git config user.email "emdashbot[bot]@users.noreply.github.com" | ||
| git checkout -B "$BRANCH" | ||
| git add -A | ||
| git commit -m "$MSG" || { echo "pushed=false" >> "$GITHUB_OUTPUT"; exit 0; } |
Contributor
There was a problem hiding this comment.
[needs fixing] git commit -m "$MSG" expands the agent-generated commit message in double-quoted context, so backticks or $(…) in the message execute under the runner shell. The message is derived from issue text and model output that an attacker can influence. Write the message to a file and use git commit -F instead.
Suggested change
| git commit -m "$MSG" || { echo "pushed=false" >> "$GITHUB_OUTPUT"; exit 0; } | |
| jq -r '.commitMessage // ("fix: address #" + (env.ISSUE_NUMBER))' /tmp/inv-result.json > /tmp/commit-msg.txt | |
| git checkout -B "$BRANCH" | |
| git add -A | |
| git commit -F /tmp/commit-msg.txt || { echo "pushed=false" >> "$GITHUB_OUTPUT"; exit 0; } |
| * The caller runs the classifier, then calls `resolve(state, event, arg)`. | ||
| * 5. Else (no eligible commands) -> noop. | ||
| * | ||
| * `allowDefault` is the caller's "this is a bot-authored PR" signal. |
Contributor
There was a problem hiding this comment.
[suggestion] parseMention returns '' for a comment that is just @emdashbot (mention plus whitespace). resolveComment then routes that empty string to the classifier, which fails its minLength(1) schema and falls back to none, producing a confusing silent no-op. Short-circuit empty text to a status reply.
Suggested change
| * `allowDefault` is the caller's "this is a bot-authored PR" signal. | |
| function resolveComment({ labels, body, actor, allowDefault }) { | |
| const text = parseMention(body); | |
| if (text === null) return { kind: "noop", reason: "no @emdashbot mention" }; | |
| if (text === "") return { kind: "readonly", state: currentState(labels), event: "status" }; | |
| const cmd = parseCommand(body); |
| @@ -1,18 +1,26 @@ | |||
| // Lightweight classifier shared between investigate and classify-reply | |||
| // workflows. Uses kimi-k2.7-code via our Cloudflare AI Gateway -- cheap and | |||
| // workflows. Uses kimi-k2.6 via our Cloudflare AI Gateway -- cheap and | |||
Contributor
There was a problem hiding this comment.
[suggestion] This header still says the classifier uses kimi-k2.6, but the default model below is now cf-wai/workers-ai/@cf/qwen/qwen3-30b-a3b-fp8.
Suggested change
| // workflows. Uses kimi-k2.6 via our Cloudflare AI Gateway -- cheap and | |
| // Lightweight classifier shared between investigate and classify-reply | |
| // workflows. Uses qwen3-30b via our custom cf-wai Workers-AI-over-gateway | |
| // provider -- cheap and fast for structured classification tasks. |
| // 2. Every non-terminal state has at least one outgoing transition. | ||
| // 3. No dead ends: every state can reach a terminal, and every terminal | ||
| // has a `reopen` edge back into the live machine. | ||
| // 4. Every state is reachable from the `triage` entry state. |
Contributor
There was a problem hiding this comment.
[suggestion] The comment says the entry state is triage, but ENTRY_STATE is unmanaged (triage is the landing state after reopen/hand_back).
Suggested change
| // 4. Every state is reachable from the `triage` entry state. | |
| // 4. Every state is reachable from the `unmanaged` entry state. |
github-actions
Bot
added
review/awaiting-author
oxlint: - Hoist regex literals to module scope (router.cjs, router.test.cjs, generate.ts). - Switch hasOwnProperty -> Object.hasOwn (router.cjs). - Use Set.has() instead of Array.includes() for label sets (router.cjs, tests). - Rename shadowed loop variable in machine.ts. - Centralize Object.keys() narrowing into typed helpers in generate.ts; add targeted oxlint-disable comments in machine.ts where the narrowing is guaranteed safe by the Record<StateId, ...> declaration. - Remove unused VERB_RE constant. zizmor: - Add persist-credentials: false to checkouts in bot-linter, bot-machine-check, and orchestrate (they only need read access; the app token does every write). - Scope create-github-app-token permissions explicitly in both orchestrate (issues/PRs/actions write, contents read) and investigate-run (contents write for the branch push, plus issues/PRs). - Document that investigate-run's checkout intentionally persists the app token for the bot/fix-<n> push step; ignore the artipacked finding for that file. - Ignore the orchestrate pull_request_target finding with justification: that job only reads PR/issue metadata via github-script and never executes PR code. Verified locally: oxlint clean on .github/, 23 router tests pass, generate --check clean, zizmor clean on all four new workflows.
machine.json and BOT_STATE_MACHINE.md are produced by .github/bot/generate.ts (`JSON.stringify(..., "\\t")` for the json, template-string assembly for the md). The bot-machine-check workflow's --check mode does a byte-for-byte compare against the on-disk files to catch drift from the spec. oxfmt was reformatting both (e.g. collapsing short arrays in machine.json onto one line), so the auto-format step would rewrite them and the next CI run would then fail the drift check with a misleading 'artifacts are stale' message. Add both paths to .oxfmtrc.json's ignorePatterns -- same pattern the repo already uses for packages/registry-lexicons/src/generated/**. Drift check is now stable: the generator is the only thing that writes those files, and CI compares like-with-like.
github-actions
Bot
added
review/needs-rereview
…ioning Blockers raised by emdashbot's review: 1. Shell injection on commit message (investigate-run.yml). git commit -m "$MSG" expanded the agent-produced commit message in double quotes, so backticks or $(...) in the message executed under the runner shell. The message is derived from issue text and model output that an attacker can influence. Now: write the message to a file via jq with --arg and pass it via git commit -F. 2. Flue 1.0 migration of legacy workflows. Beta.5's flue CLI dropped --payload (replaced by --input). investigate.yml, reporter-reply.yml, and maintainer-reply.yml all still invoked --payload, so they would have errored on every run after merge. Migrated all three to --input. 3. V2 double-fire risk. With BOT_STATE_MACHINE_V2 on, the legacy investigate.yml and the new orchestrate.yml would both react to the bot:repro label, racing label flips and spawning duplicate agent runs. Added vars.BOT_STATE_MACHINE_V2 != '1' guards to the three legacy workflows (investigate, reporter-reply, maintainer-reply) so they are no-ops once V2 is enabled. orchestrate.yml owns those paths under V2. 4. Executor missing runner provisioning. investigate-run.yml shipped with a placeholder comment that elided pnpm/root-deps/Playwright/agent- browser/package-builds 'for review clarity'. Carried those steps over verbatim from investigate.yml so the agent sandbox has the toolchain the repro-admin/repro-public skills need. Smaller items: 5. Empty @emdashbot mention. router.resolveComment now returns a readonly status decision when the mention text is empty (mention plus whitespace), short-circuiting before the classifier so it never silently returns 'none'. Added a test. 6. Stale docs. classifier.ts header (kimi-k2.6 -> qwen3-30b via cf-wai) and machine.ts invariant comment (triage -> unmanaged entry state). Also re-applies the lint cleanups (regex hoisting, Set vs includes, Object.hasOwn, centralized typed-key casts) that were lost when the auto-format bot's commit was resolved with --ours in the previous rebase. Verified locally: oxlint clean, zizmor clean on new workflows, 24 router tests pass, generate --check clean, YAML parse for all touched files.
Contributor
Overlapping PRsThis PR modifies files that are also changed by other open PRs:
This may cause merge conflicts or duplicated work. A maintainer will coordinate. |
Found by an adversarial second-opinion review pass. Blockers (RCE / token exfil / data corruption): 1. App token exfiltration via persisted git credentials. The agent ran in the same workspace where actions/checkout had persisted the app token, so a prompt-injected agent could read .git/config (or just `git push` with the persisted token) despite supposedly only getting the read-only AGENT_GH_TOKEN. Fix: persist-credentials: false in checkout; install the token transiently into the remote URL just for the trusted push step (.github/workflows/investigate-run.yml). 2. Orchestrator could not dispatch the executor. The brain's app token was scoped permission-contents: read, but GitHub maps POST /repos/.../dispatches to contents:write, not actions:write. Every transition with a repository_dispatch action would 403 AFTER labels had already flipped, leaving issues stuck in bot:working with no executor running. Fix: contents: write on the brain token (.github/workflows/orchestrate.yml). 3. Classifier arg newline injection into $GITHUB_OUTPUT. The bash step echoed `arg=$(jq -r '.arg' ...)` straight into $GITHUB_OUTPUT. Multi-line model output would inject additional event=... lines and override the deterministic event chosen earlier in the same file. Fix: drop the shell roundtrip entirely; the apply step now reads /tmp/classify-result.json with fs.readFile and JSON.parse. The event is also whitelisted against the machine's known set as belt + braces. 4. fixed: true with pushed: false still transitioned to awaiting_feedback. outcomeFromResult ignored the push step's report, so a model claim of 'fixed' with no actual branch (no diff staged, push rejected, or human commits we refused to clobber) asked the reporter to confirm a branch that didn't exist. Fix: wire steps.push.outputs.pushed into outcomeFromResult; fixed without pushed=true demotes to agent.failed. Majors: 5. One-kind-one-state invariant impossible for V2 entries. The router only added a state label; nothing assigned a kind label. The linter would then flag every V2-entered issue as missing a kind. Fix: per-event defaultKind on entry events (repro -> bug, implement -> enhancement, decline -> task). resolve() now returns addLabels (plural) including the kind on transitions from unmanaged. 6. Multiline MENTION_RE let destructive bare verbs through. The /m flag made $ match end-of-line, so '@emdashbot decline\nplease no' parsed as a bare 'decline' and bypassed the destructive-event guard. Fix: capture extends to end of input ([\s\S]*, no $ anchor), so any text on a later line disqualifies the bare-verb match and the classifier sees the whole comment. 7. PR create failure still transitioned to in_review. Wrapped in .catch(warn) but state had already flipped. Fix: openPr action now awaits the create call and aborts the transition (no label flip, no comment claiming 'Moving to in_review') on failure. 8. --force-with-lease still clobbers human commits on bot/fix-N. Lease only protects against changes AFTER our fetch; a rescue commit pushed BEFORE we ran is fetched and then overwritten. Fix: before push, inspect 'origin/main..origin/bot/fix-<n>' and refuse if any commit has an author other than emdashbot[bot]. 9. Stale label snapshot in Apply step. The route step's labels were used for resolve + remove, but a cross-workflow write (executor or human) between fire and apply would race. Fix: re-fetch issues.get in apply and resolve / remove against the live label set. Minors: - Actor check now runs BEFORE the readonly return, so a drive-by @emdashbot status from a random user doesn't make the bot reply. Empty-mention path also gates on actor. - Sweep harness reports both avgMs (success-only) and avgMsWallClock (timeouts count as the full timeout budget) so a flaky model can't look 'fast' just because its timeouts are excluded from the average. - README's 'atomic remove+add' replaced with 'live re-fetch + swap'; app.ts cf-wai docstring corrected to include the workers-ai/ prefix. Verified locally: 26 router unit tests pass (added regressions for multiline bypass, fixed-without-pushed, and unmanaged kind assignment); oxlint clean; zizmor clean on all four new workflows; flue typecheck + build clean; all YAMLs parse.
Blocker: - App token still leaked via `git push` argv. A prompt-injected agent can leave a `ps -eo args` scraper running as a background process; the trusted push step's command line includes the token in the remote URL. Fix: contain agent-spawned processes before any write-token step (SIGTERM then SIGKILL anything still anchored under $GITHUB_WORKSPACE, plus dev-server ports), and switch push auth to GIT_ASKPASS so the token stays in env and never on argv (investigate-run.yml). Majors: - V2 acted on PRs labeled `bot:repro`. The `issues` event fires for PRs too, and the V2 route path didn't skip them. Fix: explicit skip in orchestrate.yml when context.payload.issue.pull_request is present. - Fork PRs named `bot/fix-N` were trusted as bot PRs. Any external contributor could open a fork PR from a branch named bot/fix-123, and comments / merges on that PR would route as feedback for issue #123. Fix: require pr.head.repo.full_name === `${owner}/${repo}` on both the pr_comment and pr_merged paths. - Label-swap TOCTOU + swallowed removeLabel errors. The remove call had .catch(() => {}); a transient API failure left the issue with two state labels and no signal that anything went wrong. Fix: don't swallow removeLabel errors -- 404 is fine (concurrent removal), any other failure throws and surfaces. The linter on its next sweep will catch a half-applied swap as expected. - Dispatch still happened AFTER state flip. Same risk class as the PR-create-before-flip from cycle 1. Fix: in orchestrate.yml's Apply step, dispatch the executor BEFORE flipping labels; on dispatch failure, abort with state unchanged and post a retry hint. Minors: - defaultKind didn't replace a mismatched existing kind on entry. An issue carrying bot:enhancement that got `@emdashbot repro` ended up bot:enhancement + bot:working, despite repro explicitly meaning "treat as a bug". Fix: on entry from unmanaged with a defaultKind, the verb wins -- replace any non-matching kind label with the event's defaultKind. Added regression tests. - Bot context filter accepted any account with user.type === 'Bot' (Dependabot, Codecov, preview bots). Fix: filter to login === 'emdashbot[bot]' so the classifier sees OUR bot's last message when resolving references like "option A". Verified locally: 29 router unit tests pass (added 3 entry-kind cases); oxlint clean; zizmor clean on all four new workflows; flue typecheck + build clean; all YAMLs parse.
Blocker:
- GIT_ASKPASS still exposed APP_TOKEN to escaped agent processes via
/proc/<git-pid>/environ. A prompt-injected agent could detach a process
with `setsid` outside $GITHUB_WORKSPACE and survive the containment
step, then poll the git process's environment during push. Containment
on the same runner is fundamentally an arms race when the agent is also
on that runner.
Fix: split investigate-run.yml into two jobs on SEPARATE runners:
- investigate (untrusted): runs the agent with only AGENT_GH_TOKEN
(read-only). Exports the agent's staged changes as a patch
artifact. NO app token reachable from this job at all.
- apply (trusted, needs investigate): fresh runner. Generates the
write-scoped app token. Downloads artifacts, validates the patch
with `git apply --check`, applies it, pushes via GIT_ASKPASS.
The agent's processes are on a different runner that has been
decommissioned by the time the token exists.
Majors:
- Cross-key concurrency race between PR-number and issue-number
concurrency groups. Can't be fixed in workflow-level expressions (the
anchoring issue # is derived from the API payload at runtime). Mitigated
by the live-label re-fetch + resolve in Apply: the second-to-race run
sees the new state and resolve() returns noop. Documented the residual
millisecond-scale TOCTOU window in orchestrate.yml.
- Same-repo human branches named bot/fix-N were still trusted. A maintainer
could create a PR from a same-repo branch with that pattern and route
comments/merges to the wrong issue. Fix: require pr.user.login ===
'emdashbot[bot]' in addition to same-repo + branch pattern.
- Dispatch-before-flip could orphan the executor's agent run. If dispatch
succeeded but the label flip failed, the executor returned with the
issue still in (say) bot:blocked, and router.resolve dropped the
agent.* outcome because the transition only exists from bot:working.
Fix: in investigate-run.yml's Apply step, if labels don't include
bot:working and the event is agent.*, synthesize effective labels with
bot:working for the routing decision. The remove list already covers
the orphaned state label because decision.removeLabels is built from
STATE_LABELS.
- Executor still swallowed removeLabel errors with .catch(()=>{}). Mirror
the cycle-2 orchestrate.yml fix: ignore only 404, throw on anything else.
- Branch ownership guard failed open on git fetch / git log failure: an
unreachable remote object meant NON_BOT="", then the push proceeded
uninspected. Fix: set -euo pipefail; explicit error if fetch or log
fails when REMOTE_TIP is non-empty (fails closed).
Minor:
- Dispatch-failure recovery comment told users to `@emdashbot retry`,
but retry is not a valid verb from unmanaged. Fix: "please repeat the
command" is verb-agnostic and works from any source state.
Verified locally: 29 router tests pass; oxlint clean; zizmor clean on
all four new workflows; flue typecheck + build clean; all YAMLs parse.
Majors: - PR-comment vs issue-comment race for the same anchoring issue. The workflow-level concurrency expression couldn't derive the anchor (PR head ref is only in the runtime API payload), so PR #200 events and issue #123 events for the same item went to different groups. Fix: split orchestrate.yml into two jobs. The route job (no concurrency) computes the anchor and emits it as an output. The apply job (which runs the classifier and does all writes) sets job-level concurrency to `orchestrate-apply-${{ needs.route.outputs.anchorIssue }}`. A PR comment for PR #200 anchored to #123 and an issue comment for #123 both land in `orchestrate-apply-123` and serialize properly. - Branch ownership guard trusted forgeable %an. A human can commit with `--author='emdashbot[bot]'` and the guard would let the bot force-push over it. Fix: stop trying to prove provenance from git metadata. The only safe rule is "don't force-push if the branch already exists". For repro/implement modes, if origin/bot/fix-N already exists we now abort with pushed=false. Maintainers who want a fresh swing can delete the branch manually. Revise mode is unaffected because it checks out the existing bot/fix-N and applies a patch on top (additive, not overwriting), so the absence of force-creation in that flow is the right behavior already. Minor: - The apply-job checkout comment claimed `persist-credentials:true here is safe`, but the code has `persist-credentials: false`. Fixed the comment to match the (correct) code, citing the GIT_ASKPASS path so a future maintainer doesn't 'correct' it back. Verified locally: router tests pass (29); generate --check clean; oxlint clean; zizmor clean on all four new workflows; all YAMLs parse.
Contributor
There was a problem hiding this comment.
The V2 state-machine direction remains the right call, and the prior blockers are largely addressed: investigate-run.yml is now a fully provisioned executor with a safe two-runner trust split, the legacy workflows are correctly gated with BOT_STATE_MACHINE_V2 != '1', the Flue 1.0 migration is consistent (--input everywhere), the empty-mention edge is routed to status, and generated docs line up with the spec. I read the current tree statically; I did not run tests, lint, or the Flue CLI.
The remaining concern that would make me uneasy about merging as-is is that investigate.yml — the still-live path until the flag is flipped — keeps the same shell-injection pattern the previous review fixed in investigate-run.yml. A pre-existing bug in a touched file is still a live exploit, so it should be fixed before cutover. The other findings are functional gaps / least-privilege nits in the new code.
What’s fixed vs. still open:
- ✅ Executor provisioning: pnpm, Node, browser stack, root deps, package builds, Flue build all present.
- ✅ Backwards-compat guards:
investigate.yml,reporter-reply.yml,maintainer-reply.ymlskip when V2 is on. - ✅ Flue 1.0 CLI contract: old workflows use
--input, not the deprecated--payload. - ✅ Safe commit path in the new executor:
git commit -F /tmp/commit-msg.txt. - ✅ Empty
@emdashbotmention routes to read-only status instead of failing the classifier schema. ⚠️ Legacyinvestigate.ymlstill expands an AI-generated commit message ingit commit -m "..."(shell injection).⚠️ investigate-run.ymlrefuses to overwrite an existingbot/fix-<n>for any mode other thanrevise, which can spuriously failretry/repro/implementtransitions.⚠️ orchestrate.ymlmay leave a bot PR open whendeclineis issued from the issue rather than the PR.⚠️ orchestrate.ymlapply job requestsactions: writewith no apparent need.
Findings
-
[needs fixing]
.github/workflows/investigate.yml:553git commit -m "$COMMIT_MSG"expands the model-generated commit message inside double-quoted shell text, so backticks,$(...), or$VARinside the message execute as commands on the runner. The previous review fixed this exact pattern ininvestigate-run.ymlby committing from a file; the legacy workflow is still the live path untilBOT_STATE_MACHINE_V2is flipped, so the injection remains exploitable now. Write the message to a file first and commit with-F.printf '%s\n' "$COMMIT_MSG" > /tmp/commit-msg.txt git commit -F /tmp/commit-msg.txt || { echo "::warning::no staged changes to commit on $FIX_BRANCH" } -
[needs fixing]
.github/workflows/investigate-run.yml:258-264The branch-exists guard refuses to push for any mode other than
revise, but the machine dispatchesinvestigate.reprofrom several transitions wherebot/fix-<n>may already exist:retryfromawaiting_feedbackorfailed/blocked, a secondimplementon a previously attempted issue, etc. In those casespushed=false,outcomeFromResultdemotes the result toagent.failed, and the issue lands inbot:failedeven though the branch is bot-owned. The guard is too conservative—either verify branch ownership by ancestry/committer metadata, or exempt legitimate overwrite transitions so retries do not spuriously fail. -
[needs fixing]
.github/workflows/orchestrate.yml:326-329closePronly runs whenp.prNumberis set, which is only populated for PR comments. If a maintainer declines from the anchoring issue (@emdashbot declineon the issue while it isin_review), the issue label flips tobot:declinedbut the bot PR stays open. The apply step should look up the openbot/fix-${p.issueNumber}PR and close it whenprNumberis missing. -
[suggestion]
.github/workflows/orchestrate.yml:221The apply job requests
permission-actions: write, but it only firesrepository_dispatch(which GitHub maps tocontents:write) plus issues/PR writes. There is no Actions API call, so this scope is unnecessarily broad and will likely trigger least-privilege warnings (e.g., from zizmor). Remove it unless a specific future transition needs it.permission-contents: write permission-metadata: read permission-issues: write permission-pull-requests: write
github-actions
Bot
added
review/awaiting-author
…trands the issue If `bot/fix-N` is deleted between dispatch and execution, both investigate and apply jobs previously failed at their own checkout step (ref: bot/fix-N). The issue then sat in bot:working forever because the result-application step never ran. Fix: always check out main first, then attempt to switch to bot/fix-N as a separate non-failing step. If the fetch/checkout fails, the agent run proceeds against main (typically producing not-reproduced / failed), and the trusted apply job still reaches its result-application step which transitions the machine state out of bot:working.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-do | f73cef6 | Jun 25 2026, 02:31 PM |
github-actions
Bot
added
review/needs-rereview
Copilot stopped reviewing on behalf of
ascorbic due to an error
June 25, 2026 14:07
Majors:
- reject drops feedback. The reject event had no `arg: "feedback"` so
the classifier never extracted the reporter's free-text complaint into
retryContext. Fix: add arg: "feedback" so the comment body flows into
the dispatched revise run.
- retry lies about "re-run the last agent action". The transition table
hardcodes every retry -> investigate.repro; we don't persist the
previous mode. Fix: update the description to match what it actually
does ("Re-run the bug reproduction pipeline"). After implement/revise,
re-issue the original command.
- PR merge events were lost while the issue was bot:working. A revise
comment moves to working; if the maintainer merges the bot PR before
the revise agent returns, working+pr.merged had no transition and the
merge was silently dropped. Fix: add working -> done on pr.merged. The
late agent.* result that follows is intercepted by the new terminal-
state guard below.
- Agent results overrode deliberate human state changes. The orphan-
recovery synthesized bot:working for ANY non-working state with an
agent.* event, including bot:human-owned (after `@emdashbot take
over`) and bot:declined / bot:done. Fix: when the agent result lands
on a terminal or human-owned state, ignore it with an explanatory
comment. Only synthesize bot:working over non-terminal, non-human
states (the real dispatch-flip-failed case).
- Conflicting state labels stranded the item. If addLabels succeeded but
removeLabel threw non-404, currentState() returned null and every
subsequent mutating command returned noop. The linter only reported;
it didn't repair. Fix: add a destructive maintainer-only `reset`
verb that fires from ANY state including conflict, strips every state
label, and lands on triage. Bare-verb-only (excluded from classifier
candidates) so a misread sentence can't reset state.
- Bot context only read the first 100 comments. On long-thread issues,
the latest emdashbot[bot] comment wasn't on page 1, so the classifier
resolved "option A / option B" references against an older bot
message. Fix: paginate via github.paginate.iterator and keep the most
recent emdashbot[bot] body.
Minor:
- Failed-push could still post "Fix applied". When the trusted push set
pushed=false (branch exists, patch conflict, push 5xx), the comment
still used result.notes (which can be the agent's success text), so
the bot posted "Fix applied: ..." with a failed footer. Fix: when
claimedFix && !pushed, override the body with the push failure reason.
Blocker: - Agent patch could modify same-repo .github/workflows files. The trusted apply job pushed the agent's patch wholesale to bot/fix-<n>; Actions runs branch workflows with same-repo event semantics, so a prompt- injected agent could add .github/workflows/exfil.yml or modify an existing workflow to print/exfiltrate secrets. Fix: before applying the patch, run `git apply --numstat` to list touched paths and refuse any path matching ^.github/(workflows|actions)/. A bona fide CI change has to come from a human PR. Majors: - Late agent results overrode `@emdashbot reset`. The cycle-6 inert set covered done/declined/human_owned but not triage (the state reset lands on). A maintainer who reset mid-run would see the late agent.* result synthesize bot:working and undo the reset. Fix: include triage in inert. - pr.merged no-oped from awaiting_feedback. Cycle 6 added working->done but missed the case where a revise produced a new fix and the PR is still open while waiting for confirm. Fix: add awaiting_feedback->done on pr.merged. - reset preserved a mismatched kind label, and the next entry verb didn't reassign it. defaultKind reassignment only fired from unmanaged; after reset the issue carries bot:triage, so from there left a stale bot:enhancement in place. Fix: trigger defaultKind reassignment from both unmanaged AND triage entries. Added regression test. - Classifier could return arg-required events with no arg. The valibot schema makes arg optional, so the model could pick without extracting the feedback (or without the directive), and the dispatched revise/implement run would lose context. Fix: in orchestrate's apply step, when the chosen event's meta.arg is set but cls.arg is empty, default arg to the comment text. The comment IS the feedback in those cases. - openPr could succeed while the label flip failed transiently, leaving the issue in the old state with a real PR open. Fix: retry the label flip up to 3 times with backoff. If it still fails, post a recovery comment instructing the maintainer to `@emdashbot reset` (now an available verb since cycle 6) and re-issue the command.
ascorbic
added a commit
that referenced
this pull request
Jul 20, 2026
* infra/emdash-bot: scaffold Cloudflare-target Flue app (Phase 0 spike)
The state-machine work from the closed feat/bot-state-machine branch is
being rebuilt on Cloudflare per .opencode/plans/0003-bot-on-cloudflare.md.
Phase 0 is the integration spike: stand up a Cloudflare-target Flue app
with the container-backed Sandbox integration and Workers AI binding,
prove the build succeeds, then validate end-to-end locally.
This commit lands the scaffolding:
- package.json / tsconfig.json mirroring infra/flue-review.
- wrangler.jsonc: AI binding, Sandbox + FlueRegistry DOs, container
declaration, R2 workspace bucket. No Orchestrator DO yet (Phase 3).
- Dockerfile: minimal base for the spike (Node + pnpm + git + gh). The
full toolchain (Chromium, agent-browser) lands in Phase 2 when we
port the repro skills.
- .flue/cloudflare.ts: re-exports Sandbox DO, narrows Env.Sandbox type.
- .flue/app.ts: Hono root with Flue's standard /workflows routing.
- .flue/lib/classifier.ts: the shared classifier agent, defaulting to
qwen3-30b-a3b-fp8 via the cloudflare/* binding-native provider id
(no API key in scope).
- .flue/workflows/classify-command.ts: state-aware free-text classifier,
ported from the previous attempt with cf-wai/* model strings replaced
by cloudflare/@cf/* binding strings.
- .flue/workflows/investigate.ts: minimal stub agent that runs inside
the container-backed Sandbox via cloudflareSandbox(getSandbox(...)).
Returns structured classification only -- real five-stage pipeline
lands in Phase 2.
Verified: `pnpm typecheck` and `pnpm build` clean. Both workflows
discovered. Sandbox container declaration parses. The Flue beta.5 +
@cloudflare/sandbox 0.12.1 + agents 0.14.5 + hono 4.12.23 dependency
set installs and resolves cleanly under the workspace's release-age
policy.
* infra/emdash-bot: Phase 0 spike validated end-to-end
Both halves work locally:
classify-command (qwen3-30b via env.AI binding):
POST /workflows/classify-command?wait=result
-> {event:'retry', tokens:{input:1741,output:292}, model:'cloudflare/@cf/qwen/qwen3-30b-a3b-fp8'}
investigate (glm-5.2 inside Cloudflare Sandbox container):
POST /workflows/investigate?wait=result
-> {kind:'bug', area:'admin', requiresBrowser:true, tokens:{input:1624,output:60},
model:'cloudflare/@cf/zai-org/glm-5.2'}
The structural credential boundary works as designed: no API key is in the
agent's process address space at any point. The whole class of bugs the
previous attempt's eight review cycles patched is gone.
Fixes since first scaffold:
- Dropped process.env references from the agent definitions (Workers have
no process.env; Flue rejected the AgentDefinition shape at runtime).
- Dockerfile: switched to a multi-stage build that COPIES Node + npm + pnpm
from the official node:22.21.0-bookworm-slim image rather than downloading
inside the sandbox base. The Cloudflare Sandbox base image has a broken
CA bundle that fails curl -fsSL against nodejs.org with 'self-signed
certificate in certificate chain'; copying across avoids any TLS in the
final image. Symlinks for npm/pnpm CLIs get rebuilt because Docker COPY
flattens slim's symlinks; chmod +x on the .cjs targets because they lose
their executable bit during COPY.
- wrangler.jsonc: added Flue's workflow DO classes (FlueClassifyCommand-
Workflow, FlueInvestigateWorkflow) plus FlueRegistry to new_sqlite_classes.
Flue auto-adds the durable_objects.bindings in its merged .flue-vite.
wrangler.jsonc but does not add the migrations; the user has to declare
them sqlite-enabled. Same pattern as flue-review's wrangler.jsonc.
Phase 0 decision gate: PASS. Browser support (Phase 2) is the remaining
load-bearing piece -- the spike only proves the agent runs in a container
with Workers AI, not that it can drive Chromium for repro-admin/public
skills. That risk gates Phase 2, not the architecture choice.
* infra/emdash-bot: browser stack works in the Sandbox container
Phase 0 stretch validation. Extended the Dockerfile with Chromium's
runtime dependencies (libnss3, libatk, fonts-liberation, etc.) and
copied bgproc + agent-browser from the node-source stage. Added a
throwaway sandbox-probe workflow that runs a sequence of shell commands
inside the container to verify each piece.
Confirmed inside the Sandbox container:
- Node 22.21.0, pnpm 11.1.3, git 2.34.1, bgproc, agent-browser 0.30.1.
- 'agent-browser install' downloads Chrome 150 (~179 MB) in ~50s.
- Chromium launches and loads pages; agent-browser open/read works.
Two cleanup items for Phase 1 (not blockers):
- Chromium reinstalls every run because each Sandbox DO call gets a
fresh $HOME. Phase 1 should either bake Chromium into the Dockerfile
(~200 MB image growth) or persist /root/.agent-browser/browsers
across runs via a volume / R2 cache.
- Locally, Cloudflare WARP intercepts TLS with a corp Zero Trust cert
the sandbox base doesn't trust, so chromium shows cert warnings on
page loads. This is dev-machine only and won't happen on real CF
edges; safe to defer.
Decision: Phase 0 done. Browser-in-Sandbox works. Phase 1 (orchestrator
DO + webhook ingress + real investigate pipeline port) can proceed.
* infra/emdash-bot: port state machine + router + 32 unit tests
Ports machine.ts and router.cjs from the closed feat/bot-state-machine
branch into infra/emdash-bot/.flue/lib/. router.ts re-exports
findTransition from machine.ts (single source of truth) and uses
TypeScript-native exports instead of the CommonJS wrapper.
Sets up vitest in tests/unit/ with the full 32-case router suite ported
from node:test. All pass in 4ms. tsconfig includes tests/ for typecheck.
These are the substrate-agnostic bits of the bot logic; the OrchestratorDO
will wrap them with per-issue serialization in the next commit.
* infra/emdash-bot: OrchestratorDO skeleton + workers-pool test rig
Adds the per-issue Orchestrator Durable Object as the source of truth for
the bot lifecycle. The DO holds state, kind, currentRunId, prNumber, and
a bounded event log; webhook events route through event() and the
intrinsic per-instance serialization eliminates the PR-comment race that
plagued the Actions-based predecessor (PR #1606 cycle 4).
This commit is the SKELETON: resolve + persist + delivery dedupe + stale-
run discard + inert-state guard. GitHub side effects (label flip, comment,
PR ops) and workflow invocation land in the next commit alongside the
webhook handler, since they share the App-token-issuing helper.
Test infrastructure also lands here. Pure router tests stay in tests/unit/
(plain vitest, ~4ms). DO tests live in tests/integration/ and run under
@cloudflare/vitest-pool-workers against a real workerd isolate, with a
separate wrangler.test.jsonc + test entry that mirrors prod bindings but
skips Flue-generated workflow DOs (not exported by the test entry). Both
suites run via pnpm test; 32 unit + 7 integration tests pass.
* infra/emdash-bot: webhook ingress -> Orchestrator DO
POST /webhook/github verifies the X-Hub-Signature-256 HMAC against the
raw body, normalizes the GitHub payload into a NormalizedEvent, and
dispatches into the per-anchor Orchestrator DO via getByName.
Handles issues, issue_comment, pull_request, pull_request_review, and
pull_request_review_comment events. The classifier hand-off lives in the
DO; the webhook resolves bare verbs deterministically (parseCommand) and
flags everything else as needsClassify. Bot-author detection for the
in_review default-comment-event is intentionally deferred (defaults to
false, routes through classifier) until we have a known bot login binding.
Tests:
- 25 unit tests for actor classification + payload normalization (pure)
- 11 integration tests under @cloudflare/vitest-pool-workers: signature
verify (the workerd-only timingSafeEqual), full SELF.fetch -> DO round
trips for the deterministic verb path, free-text classify-pending path,
duplicate-delivery dedupe, and signature/JSON rejection.
The core routes (health + webhook) live in routes.ts so the workers-pool
test entry can mount them without pulling in Flue's workflow routes
(those need workflow DOs the test wrangler does not declare). Production
app.ts still mounts both.
* infra/emdash-bot: GitHub side effects in OrchestratorDO
Ports readAppCreds + mintInstallationToken + JWT helpers from
infra/flue-review, adds bot-specific API helpers (addLabels, removeLabel,
postIssueComment, getIssueLabels), and wires them into the orchestrator's
event() path.
The transition path now: resolve -> applySideEffects (label flip + comment)
-> persistDecision. Side effects run BEFORE persist so a label-flip
failure leaves DO state unchanged and the next event retries; comment
failures are non-fatal and logged.
Tokens are cached in DO storage with a 55-minute window. Dev mode (no
GITHUB_APP_PRIVATE_KEY) cleanly skips all GitHub calls with a log line --
no exceptions, DO state still advances locally.
Adds GITHUB_OWNER + GITHUB_REPO as wrangler vars for the single managed
repo. Multi-repo support deferred; would need owner/repo plumbed through
NormalizedEvent and the DO signature.
The webhook normalizer now stamps anchorNumber onto every NormalizedEvent
so the DO can address the GitHub side. Tests stay green via empty PEM
binding -> readAppCreds returns null -> side effects skip.
* infra/emdash-bot: wire classifier into OrchestratorDO via ctx.exports
Free-text comments now invoke the classify-command workflow synchronously
via ctx.exports.default (the loopback service binding to the Worker's
default fetch handler, no service-binding config needed). The DO blocks
~1-2s on the classifier turn, then re-enters event() with the resolved
verb.
Removes the classify-pending placeholder outcome -- the DO now always
returns a concrete decision (transition / readonly / noop), where
classifier failures degrade to noop with a logged error.
Updates tests to the cloudflare:workers exports surface (cloudflare:test
is deprecated as of vitest-pool-workers v0.13).
* infra/emdash-bot: invoke investigate workflow on action transitions
Adds runAction() to the orchestrator: on a transition with decision.action
(investigate.repro / investigate.implement / investigate.revise), fetches
the issue context via the GitHub App token, generates a runId, persists
it as currentRunId, and admits the investigate workflow via Flue's
invoke().
Extends the investigate workflow to accept the orchestrator's input shape
(runId, mode, arg, issueTitle, issueBody) and call back into the right
OrchestratorDO via env.Orchestrator.getByName(...).applyAgentResult() on
completion. Phase 1 body is still a single LLM call; the 5-stage pipeline
is Phase 2.
Wires DO and Worker exports through .flue/wrangler-main.ts so wrangler
types can infer the class generics on Cloudflare.Env's DurableObjectName-
space bindings and the Cloudflare.Exports loopback service shape. The
file is type-only -- flue dev generates its own runtime entry and the
default fetch handler here just returns 500 if hit.
Drops the previous ad-hoc Env augmentation in cloudflare.ts. No tests
needed updating: the existing integration suite already exercises the
new action path (with creds-null degrading to a logged skip).
* infra/emdash-bot: post a status comment for readonly events
Status / help mentions used to be no-ops on GitHub. The orchestrator now
posts a short reply (current state + offered commands footer) when a
readonly event lands. No-creds dev mode still skips silently.
* infra/emdash-bot: cron tick via DO alarm
Adds self-arming alarms on the OrchestratorDO. The tick recovers stale
runs (currentRunId older than 30 minutes -> drop, allowing a retry) and
reconciles label drift between DO state and the live GitHub labels (the
DO is the source of truth; manual edits are healed).
The alarm self-rearms (every 60 minutes), bootstrapped from the first
event() call. applyAgentResult now clears currentRunId on completion so
stale-run detection doesn't false-positive on a finished run.
* infra/emdash-bot: wire openPr / closePr actions
Adds createPullRequest and closePullRequest helpers to github.ts, and
splits the orchestrator's runAction into runInvestigate / runOpenPr /
runClosePr. PR number is persisted to DO storage on open so close knows
what to target.
openPr will fail in Phase 1 (no fix branch exists yet -- the investigate
workflow's git push step is Phase 2). The runError surfaces in the
EventOutcome; DO state still advances.
* infra/emdash-bot: use DO state as source of truth in resolve()
The orchestrator was passing input.labels straight through to resolve(),
even when DO storage held a newer state from a prior transition. A
follow-up event (status check, retry, etc.) carried stale labels from
the webhook's snapshot and the router decided against them.
Now: if DO storage has persisted state, project it to labels via
projectLabels() and resolve against those. Falls back to input.labels
for first-time mentions where DO storage is empty.
* infra/emdash-bot: pre-clone repo + detect push in investigate workflow
The workflow now runs in two trusted phases around the agent:
1. setupSandbox(): before the agent starts, mint an installation token,
write it to /root/.git-credentials (persistent credential store), put
it in /etc/environment as GITHUB_TOKEN, configure git identity, and
clone the repo into /workspace/repo. For revise mode, check out the
existing bot/fix-N branch.
2. detectPush(): after the agent reports done, query the GitHub API for
the bot/fix-N branch. If it exists, set pushed:true in the callback.
The router's outcomeFromResult uses pushed to gate agent.fix_ready
(fixed:true without a real branch demotes to agent.failed).
The agent now defaults to kimi-k2.7-code (the code-focused model) and is
told via instructions that the repo and creds are ready.
No-creds dev mode skips clone and push detection silently; useful for
sandbox toolchain smoke tests.
* infra/emdash-bot: outbound-proxy github auth + investigate skill
Switches the credential path from 'agent sees the token' to 'agent never
sees the token'. The Sandbox subclass declares outboundByHost for
github.com / api.github.com / codeload.github.com pointing at
authenticatedGithub, which mints a fresh installation token in the
Worker runtime, injects Basic auth (x-access-token format that works for
both git smart HTTP and the REST API), and forwards upstream.
allowedHosts denies everything except github + the registry hosts the
toolchain may need (npm + githubusercontent CDNs). enableInternet=false
makes that the only path out.
The agent now pushes directly: git clone/fetch/push to github.com is
transparent and the sandbox holds no tokens. The SKILL.md says so
explicitly to discourage poking at /etc/environment etc.
Wired in the investigate skill via the with-skill import attribute. The
orchestrator's investigate workflow import is lazy so the workers-pool
test pipeline (which has no Flue build plugin) doesn't try to parse the
markdown.
* infra/emdash-bot: fix Phase 2 sandbox setup; agent now sees a populated repo
Three bugs together prevented the sandbox setup from working:
1. interceptHttps defaults to false in @cloudflare/containers 0.3.x even
though the April 2026 changelog says HTTPS interception is on by
default. Override to true on our Sandbox subclass so outboundByHost
handlers actually see github.com traffic (otherwise the proxy was
blind to git clone over HTTPS).
2. harness.shell inherits the agent's cwd (/workspace/repo), which does
not exist before the clone. Pass cwd: "/" so setup steps run from a
directory that exists.
3. The setup script was a single .join(" && ") string containing an
if/then/else/fi block, which is a bash syntax error ("if ... && else"
never parses). Split into a typed steps[] array and run each via its
own harness.shell call. Each step's exitCode is checked explicitly
and stderr surfaces in the worker log on failure.
Adds x-emdash-dry-run: 1 header support: routes.ts forwards the flag
into NormalizedEvent.dryRun, and the orchestrator skips applySideEffects
(label flip + status comment) when dry-run. The workflow still runs in
full (LLM call, sandbox setup, push attempt). Lets local smoke tests
iterate without spamming labels/comments on a real GitHub issue.
Verified: dry-run repro against issue #1042. Setup steps all exit 0,
the agent's first command finds a populated /workspace/repo, the agent
runs real grep/find investigations against the cloned source.
* infra/emdash-bot: pnpm install in setupSandbox
Adds a final pnpm-install step to setupSandbox so the agent doesn't burn
turns rediscovering that it needs deps and figuring out which install
command works. --frozen-lockfile matches the lockfile in the checked-out
tree, which may differ between main and a revise branch.
The step is non-fatal: pnpm's post-install hooks can exit 1 in the
sandbox env (e.g. on missing build tools for native modules) while still
leaving node_modules populated enough to run tests. Failure is logged
with stdout/stderr tails but doesn't abort setup. Critical steps (clone,
checkout) remain fatal.
Bumps the install timeout to 10 min; clone gets 5 min. Updates SKILL.md
so the agent knows deps are pre-installed.
* infra/emdash-bot: allow pkg.pr.new + add native build toolchain
EmDash's pnpm-lock pins @lunariajs/core from pkg.pr.new; add it to the
Sandbox's allowedHosts so pnpm install can fetch it. Without this, the
install fails with ERR_PNPM_FETCH_520 (ContainerProxy returns 'Origin is
disallowed' status 520 for any host not in allowedHosts).
Add build-essential, python3, and python-is-python3 to the Dockerfile so
node-gyp can rebuild native modules (better-sqlite3, etc.). The previous
image only had ca-certificates + browser deps, which left node-gyp
unable to compile.
Also bumped the setup step failure log slice from 800 to 4000 chars so
node-gyp's actual error message is visible; the trailing summary alone
isn't enough to diagnose.
* infra/emdash-bot: human-readable bot replies
Replaces the bot-jargon transition comments ('Moved to working on retry
(investigate.repro).') with messages a reporter actually wants to read.
agent.fix_ready now includes a pkg.pr.new install URL (the existing
preview-releases.yml workflow auto-publishes for every push to
bot/fix-*), so reporters can pnpm-add the preview directly rather than
having to clone a branch. PR stays gated on .
Readonly status replies are state-specific now too instead of dumping
the verb list, and unmapped events get no comment at all (skip-post on
empty body).
* infra/emdash-bot: bump pnpm-install timeout to 15 min
Production sandbox network is slower than local docker; the 10 min limit
killed the install on a real run against #1623 before postinstall hooks
(better-sqlite3, sharp, workerd binaries) could finish. 15 min covers
the cold-cache case based on dev timings (~7 min) plus headroom.
* infra/emdash-bot: beefier sandbox + drop unused probe DO
- instance_type: standard-3 (2 vCPU, 8 GiB, 16 GB). The default lite (256
MiB / 1/16 vCPU) was killing pnpm install on cold cache.
- NODE_OPTIONS=--max-old-space-size=6144 so node/pnpm processes can use
the increased memory ceiling (mirrors what we bumped for emdash builds).
- Drop FlueSandboxProbeWorkflow. It was a Phase 0 toolchain probe that
hasn't been invoked since; v2 migration deletes the DO class.
* infra/emdash-bot: add investigation tools to the sandbox
ripgrep, jq, tree, less, file, unzip, sqlite3. The agent fell back to
grep -rnH and similar in the first prod run because rg wasn't installed.
sqlite3 is useful when poking at D1/SQLite-backed tests.
* infra/emdash-bot: advance to failed when tick drops a stale run
A workflow can be evicted mid-run (DO hibernation, deploy rotation,
infra eviction) without producing a callback. The hourly tick already
dropped the orphaned currentRunId, but left DO state stuck on "working"
forever, with no way out except a manual reset. Synthesize an
agent.failed event after dropping the run so the state machine progresses
to `failed` and the reporter gets the failed-state comment.
* infra/emdash-bot: log agent turns via Flue observe()
Subscribes once at app startup to the Flue event stream and logs a
compact one-liner per run/turn/tool event. wrangler tail now shows what
the agent is thinking, not just the raw sandbox.exec commands -- much
easier to spot when the agent is stalled vs. mid-LLM-call.
* infra/emdash-bot: comment on agent.reproduced transitions
The agent.reproduced event maps to working -> blocked; we already had
text for the other agent.* events but missed this one. The state flipped
silently on #1589, leaving the reporter with just a label change and no
explanation.
* infra/emdash-bot: scoped GitHub API for agent; comments are agent summaries
The outbound proxy now signs api.github.com requests scoped to the
current anchor issue/PR. The agent can curl GETs anywhere on
api.github.com, POST comments and reactions on its own issue, and push
to bot/fix-<n>; writes to other issues, PRs, or repos are denied 403.
Anchor + repo are passed via ctx.params, installed per-run by the
workflow before the agent session starts.
Replaces the boilerplate bot-jargon comments with the agent's own
summary. User-driven transitions (repro / implement / confirm /
decline / reopen / reset / take_over / hand_back) no longer post --
the user just typed the verb, echoing it is noise. Agent transitions
post the summary verbatim, with structural footers (pkg.pr.new install
URL for fix_ready; "reply with steps" prompt for not_reproduced; etc.).
SKILL.md tells the agent the summary IS the comment, and to write it
to the reporter.
* feat(emdash-bot): migrate to Flue 2 agents
* fix(emdash-bot): pin Flue Hono resolution
* chore(emdash-bot): remove durability spike
* refactor(emdash-bot): harden type boundaries
* fix(emdash-bot): allow reviewed Flue nightlies
* fix(emdash-bot): address Flue 2 review feedback
* fix(emdash-bot): preserve investigation mode
* chore: add empty changeset
Merged
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Rebuilds the emdashbot orchestration layer as a single executable state machine, migrates the Flue agent to 1.0, and replaces the keyword-based command grammar with a state-aware free-text classifier. The actual agent work (reproduce / diagnose / verify / fix) is unchanged; this is all the messy plumbing around it.
The change is everything behind
vars.BOT_STATE_MACHINE_V2, so the live bot is untouched until that flag is flipped. The three current orchestration workflows (investigate.yml,reporter-reply.yml,maintainer-reply.yml) keep running as-is.What you get when V2 is on:
One executable state machine instead of label-driven YAML spread across six files.
.github/bot/machine.tsis the single source of truth (kinds, states, events, transitions, guards), andmachine.json+BOT_STATE_MACHINE.mdare generated from it (CI fails on drift). A pure-JSrouter.cjs(23 unit tests) is the only place transition logic lives.Free-text commands.
@emdashbot <verb>only fires deterministically when the mention is exactly a bare verb, so prose like"I don't think we should implement this"no longer accidentally triggersimplement. Free text routes to a single state-aware intent classifier (classify-command), constrained to the current state's offered commands minus the destructive ones (decline,take_over). Those still require an exact bare verb, so free text can never silently close or disengage an item without any confirmation-round machinery.Flue 1.0 migration of
.flue/.createAgent→defineAgent/defineWorkflow/defineAction. The investigate pipeline collapses its three agents (classifier / investigator / fix) onto one workflow agent with separate sessions and per-operation model overrides for classify and fix. Behaviour preserved, including thewithCapacityRetryhardening on every model stage.Eval suite + sweep harness. Sentry vitest-evals with a 43-case labeled
(state, comment) -> eventdataset shared between the vitest gate (evals/command-classifier.eval.ts) and a model-sweep harness (evals/sweep.ts) that records pass rate, latency, tokens, and cost from the live Workers AI price sheet. The dataset is the safety net for letting the model drive transitions.Classifier default changed to
qwen3-30b-a3b-fp8based on a sweep against the 43-case dataset:kimi-k2.7-code(was default)qwen3-30b-a3b-fp8(new)~9× cheaper, ~4.4× faster, one case behind on a 43-case suite. qwen3's misses cluster on benign idioms (
"ship the second option","give it another go") where it conservatively returnsnonerather than guess; the maintainer just re-issues the bare verb.FLUE_CLASSIFIER_MODELis env-overridable for future sweeps. Fix stage stays onkimi-k2.7-code(classifier evals don't speak to code-fix quality, that's a separate eval to design).Custom
cf-waiprovider in.flue/app.tsbecause pi-ai'scloudflare-ai-gatewaycatalog is a stale snapshot (no k2.7-code, no glm-5.2) andcloudflare-workers-aiis binding-only (Worker-only, useless from Node).cf-wairegisters an HTTP provider against the gateway/compatendpoint, so the Node bot reaches the full current Workers AI catalog while staying inside a Workers-AI-only scoped token.Three new workflows behind the flag:
orchestrate.yml— the brain. Events →resolveComment→ classifier (free text only) →resolve(state, event)→ label flip + comment +repository_dispatch.investigate-run.yml— the executor.flue run investigateon a runner → pushbot/fix-Nif fixed →outcomeFromResult→resolve→ apply. The agent only ever holds the read-onlyAGENT_GH_TOKEN; every write uses the app token.bot-linter.yml— read-only invariant check (exactly one kind + one state label per managed item). Also gated.Closes #
Type of change
(Bot infrastructure / CI. Gated behind a repo variable, no observable behavior change to the live bot until that flag is flipped, which is a deliberate decision documented in the README cutover plan.)
Checklist
pnpm typecheckpasses (.fluetypechecks; the main monorepo typecheck is unaffected, this PR doesn't touchpackages/)pnpm lintpasses (n/a, no changes underpackages/ordemos/; new files are in.flue/,.github/bot/,.github/workflows/)pnpm testpasses (or targeted tests for my change) — 23 router unit tests pass (node --test .github/bot/router.test.cjs);flue build --target nodeclean; thevitest-evalssuite is wired and runnable (pnpm run evalsagainstflue dev, see.flue/evals/).pnpm formathas been run (didn't run repo-wide format; new files are tab-indented to match the existing.flue/and.github/style)router.test.cjscovers the new strict-parse + classify path +outcomeFromResult; eval suite covers the classifier end-to-endBOT_STATE_MACHINE_V2so the live bot is unaffected until that flag is flipped.)AI-generated code disclosure
Screenshots / test output
Final verification (post-rebase):
Sweep on the 43-case dataset (the data behind the model switch):
Cutover
Everything is gated behind
vars.BOT_STATE_MACHINE_V2, so merging this is safe; the live bot is untouched until the variable is flipped. The README under.github/bot/documents the four-phase cutover (foundation → shadow → cut over → new edges) and thetriage/* → bot:*label migration map.Commit shape note
The three commits are slightly mis-split due to a rebase conflict resolution (commit 1 absorbed some files from commit 2's intended scope). The final tree is correct and verified; happy to interactive-rebase to clean up the boundaries if reviewers prefer.
Try this PR
Open a fresh playground →
A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.
Tracks
feat/bot-state-machine. Updated automatically when the playground redeploys.