[No QA] Synthetic target for Melvin agent-device mobile smoke test - #92233
[No QA] Synthetic target for Melvin agent-device mobile smoke test#92233kacper-mikolajczak wants to merge 1 commit into
Conversation
|
|
|
|
|
I drove the Android emulator with agent-device, but the SignIn screen did not render, so I couldn't verify it. What happened: The dev app launched and showed the Expensify splash screen, but never progressed to SignIn. It then surfaced a React Native "Could not load bundle" error. The Metro dev server serving the JS bundle became unavailable mid-run (port 8081 stopped responding), so the bundle couldn't load. I tried reloading and restarting the bundler, but Metro didn't come back in this run. This looks like an environment/infra issue with the Metro bundler in this CI run, not a UI problem in the PR. agent-device tooling itself worked as expected — it opened the app, captured accessibility snapshots, took screenshots, tapped elements, and collected device logs. The only blocker was JS bundle / Metro availability.
Re-run once Metro is healthy and I can retry the SignIn render check. |
|
⏱️ Ran out of time before I could finish. This usually means the task was too complex or I got stuck exploring. Please try again with a more specific request or post in #melvinbot-ai. |
|
⏱️ Ran out of time before I could finish. This usually means the task was too complex or I got stuck exploring. Please try again with a more specific request or post in #melvinbot-ai. |
|
✅ The SignIn screen renders correctly on Android. I launched the dev build on the Android emulator and confirmed the sign-in form rendered: the Phone or email input field ( |
|
🤖 Android smoke test blocked — the dev build won't start I booted the Android emulator and launched the Expensify dev build, but the app can't get past startup. It hits a I confirmed it's not a stale bundle: a clean force-stop + relaunch reproduced the same startup error every time. This is a build/dependency-alignment problem that can't be fixed or validated on the CI emulator — it needs the Android dev client rebuilt so the native Reanimated matches the JS bundle. I avoid native/dependency changes here since they can't be verified on CI. Recommended next step: rebuild the Android dev APK with — Melvin |
|
🤖 Android smoke test blocked — Metro bundler went down mid-run I drove the Android emulator with agent-device and got the dev build up to the SignIn screen, but couldn't complete the login + send-message flow because the JS bundler (Metro, port 8081) stopped responding partway through. What happened
This is an environment/infra issue with the Metro bundler in this CI run — not a problem with this PR (which is an empty synthetic target anyway). It's the same Metro instability seen in earlier runs on this PR. agent-device itself worked fine — open, snapshot, screenshot, fill, and device-log capture all behaved as expected.
Heads-up for the smoke-test design (not yet verified this run): a fresh throw-away-account sign-in completes via a 6-digit magic code emailed to the account — there's no hardcoded dev code in the open-source sign-in flow. So even with Metro healthy, automating this end-to-end will need either a pre-provisioned account/session or an inbox the automation can read for the code. I hit the bundle error before I could observe the backend's response, so treat this as a caveat to plan around rather than a confirmed finding. Recommended next step: re-run once Metro is confirmed healthy and kept alive for the whole session, and decide how the test account's magic code will be supplied. I'll retry the full login → send-message flow then. — Melvin |
|
I drove the Android emulator for this smoke test but could not complete the task. The app opened to the sign-in screen initially, but I hit two independent blockers and did not send the message to exfy-perf@callstack.com. Blocker 1 — Dev environment broke mid-run (infra). Blocker 2 — No way to sign in with a throw-away account (provisioning). Suggested fixes for the smoke-test harness:
Happy to retry once either blocker is addressed. |
|
🤖 Mobile smoke test (Android) — blocked before I could send the message Here's how far I got on the Android dev build and where it stopped:
❌ Blocked at the onboarding "What's your work email?" step. The step won't advance: neither Skip nor submitting a work email moves forward, and Android back is gated ("Please finish the setup questions to start using the app"). A persistent error is surfaced — "The 'navigation' object hasn't been initialized yet" — and the accessibility tree shows two navigation subtrees mounted at the same time (a corrupted navigation state). Because onboarding can't be completed, I never reached Home and couldn't start a chat with exfy-perf@callstack.com to send the message. This does not appear to be caused by this PR — the same stuck state reproduces after a JS reload and after a full app relaunch, and this PR has no changed files. It looks like an environment/dev-build issue rather than anything in the PR. If this reproduces locally, the issue is in the onboarding navigation-initialization path; otherwise it may be a transient dev-build/Metro hiccup worth a re-run. Happy to retry once the build is healthy. |
|
|
|
🪲 [debug] CHECK 1 (live reload): edited the sign-in screen's button label and ran |
|
🪲 [debug] CHECK 2 (onboarding nav): stuck on the "What's your work email?" step. The Skip button is fully visible and unobstructed, but pressing it (durable selector AND exact-center coordinate) no-ops every time. Root cause confirmed: the accessibility snapshot shows the onboarding modal and two full copies of the Home tab bar mounted at once, and a tap on the modal's email field instead collapsed the press-release cards on the Home screen behind it — so touches are passing through the modal to the Home navigator underneath. This is the known double-mount / detached-overlay issue on modal transitions, not the warning toast. Trying a Metro re-mount next. |
Android emulator — two-check reportCHECK 1 — live-reload iteration loop: does not work in this CI envMade a small visible edit to the sign-in screen's button label, ran
CHECK 2 — sign in + complete onboarding to Home: reached HomeSigned in with a throwaway test account and walked the full onboarding using the navigation ladder (durable Correction to my earlier debug note: I initially suspected taps were passing through the onboarding modal — that was a misread (the Home feed behind it was just loading async). The modal was interactive the whole time. The real obstacle: the dev-mode warning toasts (incl. a continuously re-firing "getSnapshot should be cached" warning) float at the bottom and overlap the Skip button while the keyboard is down, intercepting the tap. Reliable workaround that unblocked it: focus the email field to raise the keyboard, which lifts Skip above the toast band — the tap then registers cleanly. The accessibility snapshot does also show the documented double-mount during the modal transition (two Home tab-bar subtrees + the modal at once), but it did not block interaction here — the toast overlap was the actual blocker. How far each check got
|
|
🪲 [debug] Edited the sign-in button label in JS, then ran |
|
🪲 [debug] Issued a second |
Live-reload verification (Android emulator) — ❌ edit did not propagateResult: A visible JS edit to the sign-in screen did not render after What I did
Before / After
Both read Continue — the expected Step 1 (watchman) — could not fully confirmI could not run
But I could not confirm watchman is actively watching the App root or that Metro is wired to use it — which is exactly the thing that would explain the failure below. Likely cause (medium confidence)The edit is on disk and Metro re-bundles on reload, yet the changed module's new text never reaches the served bundle. That means the file-change event for the edited source isn't reaching Metro, so it serves the cached transform for that module. This is consistent with the file watcher not delivering filesystem events (a common failure for watchman on containerized CI filesystems) or Metro not actually using watchman. I couldn't pin it down precisely without Suggested next steps
|
|
🪲 [debug] Opened the dev app, but the snapshot shows only a system launcher ANR dialog ("Pixel Launcher isn't responding" / Close app / Wait) — the Expensify UI hasn't surfaced yet. Dismissing and re-opening before the live-reload test. |
|
🪲 [debug] After editing the sign-in submit label and running |
Live-reload diagnosis (Android emulator)Verdict: the edit did NOT propagate. Watchman is watching the App project root, but a JS source edit + 1. Watchman stateThe App project root is in watchman's 2. Edit → reload → render
Result: the button still rendered To rule out a screenshot-timing artifact, I fetched Metro's served bundle directly: The new string is absent from Metro's own output, so this is not an app-render lag — Metro itself served a stale bundle. The Metro log shows it re-bundled Conclusion: Metro's file watcher is not observing on-disk edits in this agent environment. Even though 3. RevertEdit reverted; Side note (not the failure)A system "Pixel Launcher isn't responding" ANR dialog repeatedly overlaid the screen (emulator under load during bundling) and had to be dismissed before each capture. A LogBox notice ( |
|
🪲 [debug] Step D ( |
|
🪲 [debug] Step E (after reset-cache restart + reload): the served bundle now contains the unique string (cold recompile picked up the edit), but the app is sitting on a blank dark screen while it downloads/executes the freshly-built dev bundle over the slow emulator link. Waiting for it to finish rendering to confirm the button text visually. |
|
🪲 [debug] Step E follow-up: even after a full app relaunch (MainActivity foreground, Metro served the rebuilt bundle), the React tree never mounted — the screen stays blank and the a11y snapshot shows only the native shell with an empty content frame. The bundle-level proof for the workaround still holds (unique string present in Metro output); the on-device re-render just couldn't be confirmed on this emulator, which was already unstable earlier in the run. |
Watcher diagnosis + reset-cache workaround test — resultsRan the full A→F sequence on the Android emulator against the SignIn submit button label. Edit used the unique marker string Results
Diagnosis
Honest caveat on Step EI could not visually confirm the new text rendering on the device. After the reset-cache restart the app stayed on a blank content frame across multiple reloads and a full relaunch — the native shell was foregrounded but the JS tree did not mount. This emulator was already unstable during the run (a launcher ANR and a worklets ScreenshotsStep D — redbox Step E — blank content after reset-cache reload: Step E — still blank after full relaunch: Bottom lineThe file watcher is healthy and sees edits immediately. Live reload is broken because Metro re-serves a cached transform of the changed module. A cache-clearing restart fixes the bundle (the marker string appears), confirming the staleness is at the Metro cache layer — though a stable emulator is still needed to verify the final on-screen render end-to-end. |
|
🪲 [debug] Launched the dev app on the Android emulator and waited for splash. Instead of the SignIn screen, an RN redbox appeared: |
Android validation result — ❌ app crashes on launch, never reaches SignInTL;DR: The dev app does not load past splash to SignIn. It throws a full-screen RN redbox on launch, so sign-in/onboarding could not be attempted. No LogBox warning overlay/toast was seen — but only because the JS runtime never initialized (a redbox preempts everything). The crash is caused by the LogBox-suppression injection itself. (1) Launch → SignIn — ❌ FAILEDOn launch, instead of SignIn, a hard RN redbox appears and stays: It never advances past this screen. The error persists across a JS reload, so it is deterministic, not a transient bundling hiccup. (A "Pixel Launcher isn't responding" ANR also popped up intermittently — that's emulator load, unrelated to the app.) (2) Sign in + onboarding → Home — ⛔ BLOCKEDCould not start. The app never reaches SignIn, so the sign-in macro and onboarding ladder had nothing to drive. Home screenshot: not reached. (3) LogBox overlays / warning toasts — no overlay seen, but NOT a passNo LogBox warning overlays or yellow/red warning toasts appeared at any point. However, this is not evidence that suppression works — the app crashed with a more severe RN redbox before LogBox (or any app JS) ever initialized, so suppression was never actually exercised. Root causeThe LogBox-suppression snippet is injected as a Metro polyfill. Metro polyfills run in the global scope before the module system is installed, so App's own startup polyfill works fine because it only touches globals ( So the very feature under test (LogBox suppression) is what breaks the launch. Suggested direction (not implemented — read-only run, as requested)Move the LogBox-silencing out of a Metro polyfill and into a module that runs after the runtime is initialized but before the main module (e.g. Metro's I did not edit code, commit, or open a PR. |
|
🪲 [debug] App launch: the SignIn screen rendered behind the dialog ("Travel and expense, at the speed of chat" + Phone/email field), no redbox. Two things to note: (a) a system "Pixel Launcher isn't responding" ANR dialog (the emulator launcher, not the app) is overlaid, and (b) a bottom toast reads "Open debugger to view warnings." with a yellow ! icon. Dismissing the ANR and continuing. |
Android re-validation: self-managed Metro + LogBox suppressionRan on the Android emulator against the dev build (JS served by the self-managed Metro). Verdict below. (1) Launch → SignIn, no redbox: PASS. The app booted past splash straight to the SignIn screen ("Travel and expense, at the speed of chat", Phone/email field, Continue) with no redbox. The run-before-main LogBox silencer injected via (2) Sign in + onboarding → Home: PASS. Signed in with a throw-away alias, created the account (Join), allowed notifications, then completed onboarding via the flow macros + navigation ladder: skip work email → "Something else" → entered name → Continue. Landed on Home (Search + Home/Inbox/Spend/Workspaces/Account tabs, "All caught up"). (3) LogBox overlays / warning toasts — IMPORTANT, read carefully:
So: the LogBox overlay is silenced as intended; there is still a lightweight DevTools "warnings" bar surfacing that console warnings exist. Flagging it so you can decide whether that bar is in-scope for the suppression goal.
Two non-blocking observations from the run
Read-only run — no code edited, nothing committed, no PR opened. |
|
🪲 [debug] Triggered the reset-cache restart + |
|
✅ Verified on web: sign-in succeeds and the app loads to the Home screen. The ▶ Session recordings (1) |
|
🤖 Split distance expense in a 1:1 DM works end to end on Android. ✅ I opened a DM, tapped + → Split expense, chose the Distance tab, entered a route (1 Market St → Fisherman's Wharf, 1.72 mi), and confirmed. The split expense posted correctly to the chat.
No console errors. Verified on standalone Android NewDot ( EvidenceDistance tab selected in Split expense with Start and Stop fields visible on mapRoute entered: Start=1 Market St, Stop=Fisherman's Wharf, distance 1.72 mi shown on mapConfirm details screen: Melvin Bot paid, participants split $0.62/$0.63, amount $1.25, distance 1.72 miles, Split $1.25 buttonDM chat showing split distance expense: 1.72 mi @ $0.725/mi = $1.25, Your split $0.62, and the other user owes $0.63 OutstandingFull DM view showing map card with route and split expense details in the chat |
|
🤖 Could not verify — web test environment is blocked (no screenshot). I couldn't reach the sign-in flow, so I can't confirm the app loads to the Home screen. The web session never got past the browser's certificate gate. What happened: This is an infrastructure/tooling blocker, not a problem with this PR (which is an empty, no-op synthetic target anyway). Workarounds attempted (all failed)
Fix needed to unblock this path: a pre-session step that installs and trusts the dev CA in the agent-browser's certificate store, or a supported way to launch its Chrome with |
|
⏱️ Ran out of time before I could finish. This usually means the task was too complex or I got stuck exploring. Please try again with a more specific request or post in #melvinbot-ai. |
|
✅ Verified — the app loads to the Home screen after sign-in on web. The NewDot Home view rendered with the primary navigation (Home, Inbox, Spend, Workspaces, Account), the action buttons (Scan receipt, Expense, Report, Distance), and Concierge content — no sign-in form or magic-code prompt. No console errors. ▶ Session recordings (1) |
|
⏱️ Ran out of time before I could finish. This usually means the task was too complex or I got stuck exploring. Please try again with a more specific request or post in #melvinbot-ai. |
|
✅ Verified — the app loads to the Home screen after sign-in (web). After sign-in, the app landed directly on the Home screen: the left-hand navigation (Home, Inbox, Spend, Workspaces, Account) is rendered with Home active, and the main content shows the expected Time sensitive, For you, and Discover sections. No login/magic-code wall and no JS console errors. ▶ Session recordings (1) |
|
✅ Verified — the App loads to the Home screen after sign-in on web (dev NewDot).
▶ Session recordings (1) |
|
✅ Verified — the app loads to the Home screen after sign-in. Signed in on web and confirmed the Home screen rendered fully (navigation tabs, ▶ Session recordings (1) |
|
✅ Verified — the app loads to the Home screen after sign-in (web). Signed in and confirmed the authenticated shell rendered correctly: the Home heading, the Home/Inbox/Spend/Workspaces/Account navigation tabs, and the Search button are all present. No console errors. ▶ Session recordings (1) |
|
🤖 Could not verify — infrastructure failure I wasn't able to confirm the app loads to the Home screen after sign-in on the Android emulator. Sign-in never completed: across two attempts, the emulator session landed in onboarding right after the test account was minted and couldn't proceed to the login/Home flow, so no Home screen was reached. This looks like a session-bring-up / onboarding issue in the test harness rather than anything from this PR (which is an empty, no-op commit). Retrying the dispatch may clear it; if it persists, the onboarding-after-mint path in the Android session flow likely needs a look. ▶ Session recordings (1) |
|
🤖 Could not verify — Android sign-in did not complete (infra failure). I could not confirm the app loads to the Home screen after sign-in. On both attempts the Android session ended with This looks like an environment/session issue rather than anything in this PR (which is a synthetic empty-commit target). The retry hit the same state, so I stopped there per the tester's runbook. Retry with |
|
✅ Verified on the Android emulator — the app loads to the Home screen after sign-in. The canonical Search button is present, the bottom navigation bar (Home, Inbox, Spend, Workspaces, Account) is fully rendered, and the Home content sections (Time sensitive, For you, Recently added) are visible. ▶ Session recordings (1) |
|
🤖 I couldn't complete this verification — the header-vs-list-entry check was never reached, for two reasons: 1. The Search list was empty. On the signed-in test account ( 2. The drive session was then lost. Interacting with the "test drive" demo triggered a full-window navigation to the Expensify marketing site (expensify.com), which dropped the app session. Re-navigating back to What I verified before it dropped: the Search/Reports page itself loads and is reachable, but it's empty for this account. To get a real pass, this needs a test account that already has at least one report/expense in Search. Happy to retry once seeded data is available. |
|
✅ Done — ran the flow on web: opened Search, returned to the Inbox, and opened the Concierge chat. The chat header title reads Concierge. One note: visually the header renders as "Expensify Concierge", but the actionable/accessible title element itself reads "Concierge" ( Screenshot of the opened Concierge chat: Full flow evidence
▶ Session recordings (1) |
|
🤖 Ran the flow on web NewDot: opened Search → returned to Inbox → opened the Concierge chat. Header title check: partial. The chat header visually displays "Expensify Concierge", not just "Concierge". The element's accessibility label is
Screenshot of the opened Concierge chat: |
|
✅ Verified on web. The chat header title reads Concierge. Steps taken:
▶ Session recordings (1) |
|
🤖 Ran the flow on web: opened Search → returned to the Inbox → opened the Concierge chat. Navigation worked end to end. One nuance on the header title: the visible header renders as "Expensify Concierge" (with a "Your personal AI agent" subtitle), not simply "Concierge". The element's accessibility label is ▶ Session recordings (1) |
|
✅ Verified — the web app loads to the Home screen after sign-in. Signed in as ▶ Session recordings (1) |
|
✅ Confirmed — the chat header title reads Concierge. On the Android emulator I opened the Search page, went back to the Inbox, then opened the Concierge chat. The header shows "Concierge" (with the "Your personal AI agent" subtitle), verified by both a text assertion and the screenshot below. ▶ Session recordings (1) |
|
🤖 Ran the flow on web: opened the Search page, navigated back to the Inbox, and opened the Concierge chat. Header title: The rendered header visually reads "Expensify Concierge" (with the subtitle "Your personal AI agent") — not just "Concierge". So the strict check "header title reads 'Concierge'" does not pass on visual inspection. One nuance worth flagging: the accessibility label on the header button element does return Steps verified
Screenshot of the opened chat: ▶ Session recordings (1) |
|
I ran the flow on NewDot web (open Search → back to Inbox → open Concierge chat). Steps completed successfully, but one detail differs from the request: The chat header title reads "Expensify Concierge", not just "Concierge". The subtitle underneath is "Your personal AI agent". (The button's accessibility label is
Concierge chat open from Inbox — header reads "Expensify Concierge"Supporting screenshotsSearch page opened from left-hand navigation
Inbox after navigating back from Search
No console errors were observed during the flow. |
|
✅ Verified — the app loaded to the authenticated home screen and the Search button is present in the bottom tab bar (alongside Home, Inbox, Spend, Workspaces, and Account). ▶ Session recordings (1) |
|
I opened the Search page, navigated back to the Inbox, and opened the Concierge chat on web. The flow worked, but the header title is not exactly "Concierge" — it reads "Expensify Concierge" (with the subtitle "Your personal AI agent").
EvidenceInbox view with Concierge chat open, showing header 'Expensify Concierge' with subtitle 'Your personal AI agent'Final screenshot of Concierge chat showing header labeled 'Expensify Concierge' |
|
|
No-op commit. This PR exists only as a stable, main-aligned target for manually dispatching the Melvin App workflow's Android mobile path, so the Rock fingerprint matches a cached build. It changes no app code and should not be merged.
be98671 to
ebaa36b
Compare
✅ Android smoke test passedSigned in to the standalone Android NewDot build, opened the Concierge chat, typed a message, and sent it. Read-back confirmed the exact typed text (
EvidenceComposer showing the typed message before sendingSent message visible in the Concierge conversation, matching the typed textNotes on run stability
|
|
🤖 Done — opened the App on web, navigated to the Concierge chat, and sent "Testing time!". Confirmed the message appears in the conversation (verified via read-back of the exact text). EvidenceConcierge chat showing the sent 'Testing time!' message |
|
✅ Done — sent "Testing time!" to Concierge on web and confirmed it landed in the conversation. The message shows in the Concierge chat thread (bubble from the test user) and the inbox preview updated to "You: Testing time!". EvidenceConcierge conversation showing sent message 'Testing time!' from the test user, with inbox list preview updated to 'You: Testing time!' |
2 participants
Explanation of Change
Synthetic, no-op PR. Do not merge.
This PR contains a single empty commit and changes no app code. It exists only as a stable,
main-aligned target for manually dispatching the Melvin App workflow's Android mobile path (agent-device PoC, Expensify/melvin#172).Because it sits exactly on top of
mainwith an empty tree diff, its Rock native fingerprint matches a cacheddevelopmentDebugbuild, so the workflow'srock remote-cache downloadstep hits the cache (instead of fast-bailing on a miss). Pointing the manual dispatch at this PR gives the run a real target for its result comment without polluting any real PR.Fixed Issues
$
PROPOSAL:
Tests
N/A - empty commit, no app code changed. This PR is a workflow dispatch target only.
Offline tests
N/A - no app code changed.
QA Steps
N/A -
[No QA]synthetic target; no app code changed and this PR must not be merged.PR Author Checklist
### Fixed Issuessection aboveTestssectionOffline stepssectionQA stepssectiontoggleReportand notonIconClick)src/languages/*files and using the translation methodSTYLE.md) were followedAvatar, I verified the components usingAvatarare working as expected)StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))npm run compress-svg)Avataris modified, I verified thatAvataris working as expected in all cases)Designlabel and/or tagged@Expensify/designso the design team can review the changes.ScrollViewcomponent to make it scrollable when more elements are added to the page.mainbranch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTeststeps.Screenshots/Videos
Android: Native
N/A - no app code changed.
Android: mWeb Chrome
N/A
iOS: Native
N/A
iOS: mWeb Safari
N/A
MacOS: Chrome / Safari
N/A