feat(testing): add optional Jev browser and translation checks - #841
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_b9c73b9b-88bc-4862-ba71-8e645d041a1b) |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (26)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_9cd1d4d7-368b-441e-8495-48a2378d1fd5) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd5161598b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| export const browserOriginGuard = (origin) => `if (await page.evaluate(() => location.origin) !== ${JSON.stringify(origin)}) throw new Error('origin_changed');`; | ||
|
|
||
| export async function findPlaywrightCli() { | ||
| if (process.env.PLAYWRIGHT_CLI_BIN) return process.env.PLAYWRIGHT_CLI_BIN; |
There was a problem hiding this comment.
When PLAYWRIGHT_CLI_BIN is a documented repo-relative path such as ./node_modules/.bin/playwright-cli, returning it verbatim makes every live browser command resolve it from the newly created temporary cwd, so the executable is not found and the run ends as browser_command_failed. Resolve path-like overrides against the invocation or repository directory while leaving bare command names available through PATH.
AGENTS.md reference: scripts/AGENTS.md:L6-L6
Useful? React with 👍 / 👎.
| const sourceRelative = repository && path.relative(repository, sourceFile); | ||
| const sourceChanged = changed?.has(sourceRelative); |
There was a problem hiding this comment.
On Windows, git diff --name-only and git ls-files populate changed with forward-slash paths, while these path.relative() results contain backslashes. Consequently neither English nor target catalog changes match during a --base scan, and a valid changed-locale review ends with No translation pairs matched; normalize these comparison keys to Git-style separators as already done for git show paths.
Useful? React with 👍 / 👎.
|
|
||
| ## Optional Jev checks | ||
|
|
||
| See `scripts/jev/README.md` for the bounded browser helper. A task-owned plan lists permitted controls/actions and deterministic completion assertions; the helper observes a fresh snapshot before each choice and owns its isolated browser session. Use semantic checks for text meaning or qualitative requirements after ordinary assertions, and report uncertainty as unverified. Run offline plan validation first. Provider calls require explicit `--live`, a runtime-selected pinned model, credentials, and a budget. Prefer ordinary scripted checks for known fixed flows; do not add model calls to edit hooks or replace Bippy measurements. |
There was a problem hiding this comment.
This newly advertises node scripts/jev/browser.mjs, but the skill frontmatter still permits only Bash(playwright-cli:*) and Bash(./scripts/pw-session.sh:*). In Claude, the generated skill preserves that restriction, so an otherwise authorized Jev check can require an unexpected permission prompt or fail in noninteractive execution; add a narrowly scoped Node invocation to the canonical skill permissions and regenerate the compatibility copies.
AGENTS.md reference: AGENTS.md:L90-L91
Useful? React with 👍 / 👎.
| staleReplans: 0, | ||
| reason: 'not_started', | ||
| mode: baseline ? 'deterministic' : 'jev', | ||
| origin: plan.origin, |
There was a problem hiding this comment.
For checks against distinct paths or hash routes on the same origin, the live report records only the origin, making failed runs indistinguishable and omitting the actual URL acted on. Preserve the canonical plan.url in the report so route-specific failures remain diagnosable.
AGENTS.md reference: scripts/AGENTS.md:L5-L5
Useful? React with 👍 / 👎.
| try { | ||
| return JSON.parse(await fs.readFile(file, 'utf8')); | ||
| } catch { | ||
| throw new TranslationInputError('Unable to read valid JSON input'); |
There was a problem hiding this comment.
When one catalog in a multi-locale scan contains malformed JSON, this generic error reaches the CLI without identifying which of the English, target-locale, pair, or corpus files failed, forcing the user to inspect every input manually. Include a bounded repo-relative input path in the validation error so the failure is actionable.
AGENTS.md reference: scripts/AGENTS.md:L5-L5
Useful? React with 👍 / 👎.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_94e3afdb-869e-4481-9d6d-5fccbb4a6f64) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_6cc5b8de-57db-4d33-b033-b74c090063a9) |
1 participant
Add optional development-only Jev browser and translation QA helpers to the existing agent workflows. Browser plans restrict actions to fresh observed controls and use deterministic completion/persistence assertions. Transient target replacement discards the stale decision and permits at most two fresh choices within existing budgets; semantic uncertainty never passes.
Translation checks are read-only and explicitly scoped to keys/locales, with structure checks before bounded semantic calls. Provider credentials remain in the environment; calls require opt-in and a pinned model. No application source, dependencies, locale values, or profiler behavior changed.
Validation: immutable install; 62 helper tests and 41 workflow tests on Node 22.12.0; sync/check/format/diff checks; independent review. Actual Seedit settings flow recovered from initial subtree replacement, selected dark theme, and passed value/body-class assertions after reload. A separate optional wording check returned uncertain and correctly required review (3 calls, ~$0.000274). Small six-language translation pilot: all eight planted semantic defects caught, six good cases passed, one unverified, and two structural defects caught without calls. This is a pilot, not general accuracy evidence.
Based on published master; six unrelated local Seedit commits remain untouched outside this PR.
Final review validation: all 71 Node22 offline fixtures pass, including Windows changed-catalog matching, safe malformed-input diagnostics, browser cleanup failures, CLI execution flags, canonical URL assertions, and credential preflight. Vitest retains its default exclusions and excludes these separately tested Node fixtures; browser skill permissions include the scoped helper invocation. Workflow sync/check/tests and scoped formatting pass.
Note
Low Risk
Changes are confined to dev scripts, agent docs, CI, and Vitest excludes; helpers are opt-in with strict scoping and do not modify application runtime or translation writes.
Overview
Adds optional development-only Jev helpers under
scripts/jev/for agent workflows—no shipped app or locale file changes.Browser helper (
browser.mjs): JSON plans whitelist actions from Playwright snapshots, enforce deterministic completion/persistence assertions, and optionally use a pinned TypeSafe/Jev model for next-step choices or advisory semantic text checks. Live runs use an isolatedpw-session.shsession with origin guards, budgets, and stale-target replanning; offline plan validation and baseline (no API) modes are supported.Translation QA (
translations.mjs,translations-eval.mjs): read-only, explicitly scoped locale/key or pair-file review with structural checks first, then bounded semantic questions; private cache and a small labeled pilot corpus for evaluation.Workflow wiring:
browser-check/translatorroles andplaywright-cli/translateskills document opt-in use (credentials, pinned model, budgets; model verdicts never replace deterministic checks). Synced copies under.claude,.codex, and.cursor. New CI jobjev-helpers.ymlrunsnode --test scripts/jev/tests/*.test.mjs; Vitest excludes those fixtures.Reviewed by Cursor Bugbot for commit 5457880. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
New Features
Documentation
Tests