Summary
Add a runtime render check to the agent benchmark harness: after the deterministic repo-vs-rubric evaluators run, load the generated app in Altinn Studio's app preview with headless Playwright and score whether every page actually renders. This closes the one gap static evaluation can't cover — a structurally valid app whose layout crashes the renderer.
Replaces #171, scoped down after the eval-architecture migration: no app build, no Playwright MCP, no LLM-driven interaction.
Amended after implementation. Two additions turned out to fall naturally out of the same render engine and are now in scope: a render-fix loop in the benchmark runner, and the same check exposed to the agent as a preview_render_check loop tool. Both are opt-in and off by default.
Context
The benchmark harness (src/AI/agents/benchmarks/) scores generated apps by cloning the session branch and checking the repo against a structural rubric (bench_pages, bench_field_coverage, …). All checks parse layout/model JSON directly — cheap and precise, but blind to runtime failures: an unknown component type, a malformed expression, or a broken binding can pass every structural check and still crash or blank the form for the user.
Studio's app preview (/app-specific-preview/{org}/{app}, see previewPage() in packages/shared/src/api/paths.js) renders the form frontend against a mocked instance/party — no dotnet build, no localtest, no deployed app. That makes "does it render" a browser-load away, which is exactly smoke-check territory.
Goal
One new evaluator step in the benchmark runner that answers: does the generated app render in preview, page by page? Scored to Langfuse like the existing bench scores, on the same workflow trace and dataset run.
In scope
- Plain Playwright (Python, headless Chromium) driven directly from the benchmark runner — no MCP server, no LLM in the loop
- A reusable login helper for the local stack (log in once as the Gitea test user, persist Playwright
storage_state, reuse across items) — this was the main setup friction in the previous attempt, so it should be a small, documented utility
- Ensure preview reflects the session branch before loading (the agent commits to
altinity_session_<id>; check out that branch via the Designer API, mirroring what the frontend's reset/checkout flow does)
- Per-page render verification: iterate the rubric/
pages.order and load each layout (the preview hash supports selecting a layout directly), asserting the form renders and no error boundary / fatal console error appears
- Two new scores, following the existing conventions in
benchmarks/README.md:
bench_renders (boolean) — the app preview loads and the first page rendersbench_pages_render (0–1) — fraction of ordered pages that render without error
- Optional render-fix loop in the runner (
BENCH_RENDER_FIX=1, rounds via BENCH_RENDER_FIX_ROUNDS, default 1): failing pages are fed back into the agent session as a new goal and re-checked. bench_renders/bench_pages_render always describe the first attempt so results stay comparable across agent versions; the after-fix state is scored separately:
bench_render_fix_rounds (numeric) — rounds actually spentbench_pages_render_after_fix (0–1) — fraction rendering once the loop finishes
- The same engine exposed to the agent as a
preview_render_check loop tool (PREVIEW_CHECK_ENABLED=true, off by default), so a run can verify its own work after commit_session_branch. Permission-gated like other write-adjacent tools (denied in read-only mode, escalatable), requires a committed session branch, and reports itself unavailable rather than failing when the browser or stack is missing. Requires the Chromium layer in the agents image
- Failure containment: a render failure scores 0 with a comment (page id + error snippet); Playwright/browser being unavailable skips the check with a log line (opt-in via env flag, e.g.
BENCH_PREVIEW_CHECK=1), never fails the benchmark run itself
Out of scope
- Building or running the app backend (localtest, deploys) — preview only
- Playwright MCP, LLM-generated test steps, or goal-based interaction flows
- Form filling, validation, submission, or process/payment/signing flows
- CI wiring (benchmarks are run manually today; unchanged)
Acceptance criteria
Relationship to other work
Replaces #171 (end-to-end evaluation node using Playwright MCP): after the migration to Langfuse-managed evaluators there are no in-code judges, and functional correctness is covered by in-loop validation, the structural benchmark evaluators, and the per-trace LLM judges. This issue keeps the one thing that idea uniquely offered — runtime render verification — at a fraction of the cost, in the offline benchmark harness where flakiness can't pollute per-trace scores.
Summary
Add a runtime render check to the agent benchmark harness: after the deterministic repo-vs-rubric evaluators run, load the generated app in Altinn Studio's app preview with headless Playwright and score whether every page actually renders. This closes the one gap static evaluation can't cover — a structurally valid app whose layout crashes the renderer.
Replaces #171, scoped down after the eval-architecture migration: no app build, no Playwright MCP, no LLM-driven interaction.
Amended after implementation. Two additions turned out to fall naturally out of the same render engine and are now in scope: a render-fix loop in the benchmark runner, and the same check exposed to the agent as a
preview_render_checkloop tool. Both are opt-in and off by default.Context
The benchmark harness (
src/AI/agents/benchmarks/) scores generated apps by cloning the session branch and checking the repo against a structural rubric (bench_pages,bench_field_coverage, …). All checks parse layout/model JSON directly — cheap and precise, but blind to runtime failures: an unknown component type, a malformed expression, or a broken binding can pass every structural check and still crash or blank the form for the user.Studio's app preview (
/app-specific-preview/{org}/{app}, seepreviewPage()inpackages/shared/src/api/paths.js) renders the form frontend against a mocked instance/party — no dotnet build, no localtest, no deployed app. That makes "does it render" a browser-load away, which is exactly smoke-check territory.Goal
One new evaluator step in the benchmark runner that answers: does the generated app render in preview, page by page? Scored to Langfuse like the existing bench scores, on the same workflow trace and dataset run.
In scope
storage_state, reuse across items) — this was the main setup friction in the previous attempt, so it should be a small, documented utilityaltinity_session_<id>; check out that branch via the Designer API, mirroring what the frontend's reset/checkout flow does)pages.orderand load each layout (the preview hash supports selecting a layout directly), asserting the form renders and no error boundary / fatal console error appearsbenchmarks/README.md:bench_renders(boolean) — the app preview loads and the first page rendersbench_pages_render(0–1) — fraction of ordered pages that render without errorBENCH_RENDER_FIX=1, rounds viaBENCH_RENDER_FIX_ROUNDS, default 1): failing pages are fed back into the agent session as a new goal and re-checked.bench_renders/bench_pages_renderalways describe the first attempt so results stay comparable across agent versions; the after-fix state is scored separately:bench_render_fix_rounds(numeric) — rounds actually spentbench_pages_render_after_fix(0–1) — fraction rendering once the loop finishespreview_render_checkloop tool (PREVIEW_CHECK_ENABLED=true, off by default), so a run can verify its own work aftercommit_session_branch. Permission-gated like other write-adjacent tools (denied in read-only mode, escalatable), requires a committed session branch, and reports itself unavailable rather than failing when the browser or stack is missing. Requires the Chromium layer in the agents imageBENCH_PREVIEW_CHECK=1), never fails the benchmark run itselfOut of scope
Acceptance criteria
bench_rendersandbench_pages_renderappear on the workflow trace and dataset run in Langfuse alongside the existing bench scoresbench_renders=1,bench_pages_render<1, with the failing page named in the score commentBENCH_RENDER_FIX=1, a broken layout is fed back to the agent andbench_render_fix_rounds/bench_pages_render_after_fixare scored, whilebench_renders/bench_pages_renderstill describe the first attemptpreview_render_checkloop tool is denied in read-only mode, refuses to run beforecommit_session_branch, and reports itself unavailable (rather than erroring) when disabled or when the browser is missingRelationship to other work
Replaces #171 (end-to-end evaluation node using Playwright MCP): after the migration to Langfuse-managed evaluators there are no in-code judges, and functional correctness is covered by in-loop validation, the structural benchmark evaluators, and the per-trace LLM judges. This issue keeps the one thing that idea uniquely offered — runtime render verification — at a fraction of the cost, in the offline benchmark harness where flakiness can't pollute per-trace scores.