Skip to content

Review follow-ups: credential leak, phantom tool reference, Node floor, locator edge cases - #5

Merged
jochen-testingbot merged 1 commit into
mainfrom
fix/review-followups
Sep 21, 2026
Merged

jochen-testingbot merged 1 commit into
mainfrom
fix/review-followups

Conversation

@jochen-testingbot

Copy link
Copy Markdown
Contributor

Follow-ups from the scan of main requested after #4. Each item is independent.

Stacked on #4. Base is fix/w3c-capabilities-and-integration-coverage, not main, because item 5 touches the integration tests that PR introduces. GitHub will retarget this to main automatically once #4 merges.

1. Credential leak into the appium-mcp child

filterEnv stripped TESTINGBOT_KEY, TESTINGBOT_SECRET, TB_KEY, TB_SECRET — but config.ts also accepts TESTINGBOT_USERNAME and TESTINGBOT_ACCESS_KEY as fallbacks. Anyone configured via those names handed their credentials straight to the child process, contradicting the documented contract that the child gets credentials only baked into the injected hub URL.

The list is now an exported CREDENTIAL_ENV_NAMES, with two tests: one asserting nothing leaks into the spawned env (while unrelated vars still pass through), and one that reads config.ts and fails if getConfig() grows a fallback the filter doesn't cover.

2. tb_appiumEndpoint never existed

It was named in tb_listSessions' agent-facing description, so the agent was being told to call a tool that isn't registered.

What it should use: appium_session_management with action: 'list' — that's the proxied tool that actually owns mobile sessions. The description now says so explicitly, and states that mobile sessions are not listed by tb_listSessions.

Added tests/tools/tool-references.test.ts, which walks every registered description and fails on any tb_* reference that isn't a registered tool. This class of bug can't recur silently.

3. Stale comment in register.ts

Claimed a fallback tool is registered when the appium spawn fails; the catch block only logs. Comment now matches the code. (CLAUDE.md's version of this text was already accurate.)

4. Node floor

preflight() enforced 18; package.json declares >=20. Now 20, with a test at 18.20.0 to keep the runtime gate and engines.node from drifting apart.

5. waitUntil — implemented, not dropped

It was being passed by the integration tests and silently ignored, since tb_navigate's schema only accepted sessionId and url.

Implemented, because the underlying problem is real: navigateTo returns per the session's pageLoadStrategy, which on a page with slow third-party resources can leave the agent looking at a half-built DOM. tb_navigate now confirms document.readyState itself:

  • load (default) — complete
  • domcontentloaded — interactive or better, returns earlier on ad-heavy pages
  • none — return as soon as the driver's call resolves

A timeoutMs (default 30s) bounds the poll. Timeout warns rather than failing the call — the navigation succeeded and the page is usually usable, so erroring would leave the agent with nothing to act on. A probe that throws mid-navigation (destroyed execution context) is retried until the deadline rather than propagating. Five unit tests plus an integration case.

7. SessionManager was a one-shot

closeAll() latched shuttingDown on permanently, so every later register() threw "SessionManager is shutting down" — a library host that tears the tool family down and brings it back up in the same process got a silently dead manager.

The flag is now held only for the duration of the sweep (registrations during teardown are still rejected — tested), and register() restarts the reaper closeAll() stopped. Without that restart a revived manager's sessions would idle until TestingBot's own reaper billed them out.

Two existing tests asserted the old latching behaviour and have been rewritten to encode the new contract.

8. Secrets in logs

The CallTool handler logs { tool, args } wholesale. tb_type's text — whatever gets typed into a login form — was not redacted. Added args.text, args.capabilities (appium caps carry app credentials), args.remoteServerUrl (embeds key:secret), plus generic *.secret / *.token / *.accessKey.

Verified live: {"args":{"sessionId":"x","value":"#pw","text":"[redacted]"}}. tests/lib/logger.test.ts also asserts the non-secret fields stay readable — over-redacting makes the logs useless.

Only reachable under TESTINGBOT_DEBUG=true or dev mode, since the default level is error.

9. Locator and key edge cases

  • A leading digit in an id can't be backslash-escaped; CSS requires the hex form (#2fa-code → #\32 fa-code).
  • by: "class" with "btn primary" means the compound selector .btn.primary. It was producing one escaped class literally named btn primary.
  • "+" is both the chord separator and a real key, so it was unpressable. "+" and "Control++" now work.

Verification

npm run build passes: lint, format, 156 unit tests (up from 120), tsc. Integration suites still skip cleanly without the gate flags.

Follow-ups from a scan of the tool surface and session lifecycle.

filterEnv stripped only four of the six env names config.ts reads
credentials from, so a user who configured the server via
TESTINGBOT_USERNAME / TESTINGBOT_ACCESS_KEY handed those straight to the
appium-mcp child — contradicting the contract that the child receives
credentials only baked into the injected hub URL. The list now lives in an
exported CREDENTIAL_ENV_NAMES, with a test that fails if getConfig() grows a
fallback the filter doesn't cover.

tb_listSessions' agent-facing description pointed at tb_appiumEndpoint, which
has never existed, inviting the agent to call a missing tool. It now points at
appium_session_management (action 'list'), the tool that actually manages
mobile sessions. A new test walks every registered description and fails on
any tb_* reference that isn't registered.

preflight() enforced Node 18 while package.json declares >=20.

tb_navigate's waitUntil is implemented rather than ignored. It was passed by
the integration tests and silently dropped, since the schema only accepted
sessionId and url. navigateTo returns per the session's pageLoadStrategy,
which on a page with slow third-party resources can leave the agent looking at
a half-built DOM, so readyState is now confirmed directly: 'load' (complete),
'domcontentloaded' (interactive or better), or 'none' to skip. A timeout
warns rather than failing the call — the navigation succeeded and the page is
usually usable, so erroring would leave the agent with nothing to act on.

SessionManager.closeAll() latched `shuttingDown` on permanently, so every
later register() threw and the manager was silently dead. The flag is now held
only for the duration of the sweep, and register() restarts the reaper that
closeAll() stopped — otherwise a revived manager's sessions would idle until
TestingBot's own reaper billed them out.

Tool arguments are logged wholesale by the CallTool handler, so args.text
(tb_type's payload, i.e. whatever gets typed into a login form),
args.capabilities and args.remoteServerUrl are now redacted, along with the
generic *.secret / *.token / *.accessKey names.

Locator and key edge cases: a leading digit in an id needs CSS's hex escape
rather than a backslash; by:"class" with "btn primary" means the compound
selector .btn.primary, not one class of that name; and "+" is both the chord
separator and a real key, so it was unpressable.
@jochen-testingbot
jochen-testingbot changed the base branch from fix/w3c-capabilities-and-integration-coverage to main September 21, 2026 07:50
@jochen-testingbot
jochen-testingbot merged commit b5d1c5d into main Sep 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant