Port mentra ai local#3238
Draft
aisraelov wants to merge 15 commits into
Draft
Conversation
Give the fast Mastra agent an `ask_agent` tool that hands multi-step / personal-account / long-running work to the user's persistent giga-agent (the testaiassistant control plane), keeping the fast path for trivial queries. Delegation is a no-op unless AGENTS_BASE_URL + MENTRA_AGENT_API_KEY are set, so unconfigured environments behave exactly as before. - AgentBridge: control-plane REST client (message/wake/getTask), keyed on the SDK mentraUserId, forwarding POV photo + device context. - DelegationManager: registry for tasks that exceed the 2s grace window; settles once via webhook (primary) or poll backstop. - ask_agent tool + prompt routing: fast <=2s -> inline reply; slow -> the agent speaks a brief ack and the result is delivered later. - QueryProcessor.deliverFollowUp: re-enters the normal HUD/speaker output path, mirrors to the webview chat thread, and records the FINAL answer in history so the fast agent stays in sync. - Webhook receiver at /api/agent/webhook (public sub-app; verifies the shared x-api-key, no SDK token). - Predictive wake at "Hey Mentra" to warm the container during STT. - Webview chat renders agent action buttons (e.g. OAuth "Connect Gmail").
Adds a text input to the Mentra AI webview so the assistant can be tested by typing, with no glasses required, and gives typed messages fuller answers than the glasses voice/HUD path. - New `chat` channel: prompt drops device/vision/TTS/HUD sections and the hard word limit; markdown formatting encouraged. Glasses path unchanged. - QueryProcessor.processTextQuery: runs the agent (+ delegation) for a typed message, broadcasts the turn to the chat SSE stream, returns the answer. - POST /api/chat/send and the chat SSE stream move to the auth-only sub-app (get-or-create User) so chat works whether or not glasses are connected. - Frontend: input pill + circular send button (matches Parth's Figma), wired to /api/chat/send; responses render via the existing SSE stream. Renders agent action buttons (e.g. OAuth connect) on incoming messages.
Brings the Mentra AI webview in line with the finished V2 Figma and fixes a layout collision introduced by the chat input bar. - Header: adds the sparkle logo + "Mentra AI" / "Settings" title. - Settings: full V2 rebuild — Assistant (Model picker accordion, Wake word, Voice & language), Appearance (dark mode), About — with the card/section layout from the Figma. - Model picker: persists the choice to UserSettings.model and the agent honors it (MODEL_OPTIONS + resolveModel in config; QueryProcessor passes the resolved model to the agent, falling back to the default safely). - Chat input: replaces the old fixed bottom logo banner (which overlapped the input bar) with a "Say Hey Mentra or type below" hint chip; adds the decorative mic glyph inside the pill. - Home: subtitle now "Then ask a question — or type below.". - Fix: theme toggle used a stale-closure setState that couldn't revert on a failed save; switched to the functional updater.
Implements the giga-agent escalation in the local miniapp architecture
(backend + background/ui), the correct home for it — the cloud-SDK
miniapps/mentra-ai is only a reference copy.
Backend (key stays server-side):
- services/agent/delegation.ts: control-plane client (message/getTask/wake),
gated on AGENTS_BASE_URL + MENTRA_AGENT_API_KEY.
- tools.ts: `ask_agent` tool, advertised only when delegation is configured;
routes through a per-request holder so the service can surface actions + a
pending taskId out-of-band from the model text.
- agent.service.ts: threads the authed mentraUserId + holder, returns
{pendingTaskId?, actions?}; prompt.ts gains the delegation routing rules.
- agent.api.ts: passes auth.mentraUserId; adds GET /task/:id (poll) and
POST /wake (predictive warm) — both proxy the control plane so the key
never leaves the server.
Background + UI:
- MentraAgent: returns pendingTaskId/actions; pollDelegation() polls the
backend (which proxies the control plane — no webhook needed).
- QueryProcessor: on a pending delegation, speaks the ack now, then
awaitFollowUp() delivers the real answer as a second turn (chat:event +
glasses + history). Fast-delegation actions ride the AI message.
- system:open-url RPC channel → session.system.openUrl for OAuth connect
buttons; ChatInterface renders message.actions as buttons.
- Predictive wake fired at wake-word time.
Model picker, chat box (chat:ask), and chat:event already existed here.
miniapps/mentra-ai is only a reference copy of the old cloud-SDK Mentra AI. My earlier giga-agent / chat-box / V2-design commits were placed there by mistake; the real work lives in miniapps/mentra-ai-local (@mentra/miniapp). This restores the reference folder to its pre-port state (becbd8d), removing the misplaced changes. mentra-ai-local is unaffected.
📋 PR Review Helper📱 Mobile App Build✅ Ready to test! (commit 🕶️ ASG Client Build⏳ Waiting for build... 🔀 Test Locallygh pr checkout 3238 |
PR Agent Orchestrator State{
"cycle": 3,
"fixRound": 0,
"totalReviewerRuns": 3,
"consecutiveNoNewReviews": 3,
"openFindings": [],
"resolvedFindings": [],
"nitFindings": [],
"phase": "discovery",
"status": "in_progress",
"lastPair": [],
"stagnationFixRounds": 0,
"lastOpenCount": 0,
"fingerprintReopenCounts": {},
"mutedFingerprints": []
} |
Mentra AI (com.mentra.ai.local) becomes a system app and can drive other miniapps from the agent. - Host: add com.mentra.ai.local to SYSTEM_APPS so the SDK's system-only miniapps/actions APIs are permitted. - Discovery via context: the background gathers session.miniapps.list() each query and passes the installed apps + their declared actions into the agent context (next to location/notifications) — no tool round-trip to discover. - Three deferred tools: start_app, stop_app, invoke_app_action(pkg, actionId, params). The backend records a DeviceAction (like the open_url actions) and returns an optimistic confirmation to the model; the background executes them after the response via session.miniapps.start/stop + session.actions.invoke. - Prompt gains an "App Control" section listing the available apps/actions. - All gated on apps being present (the tools simply aren't offered otherwise).
Fix the split-layout pack (the generic `mentra-miniapp pack` expects a flat miniapp with a build script; ours is backend/ + miniapp/). Adopt merge's pattern: build.ts now stages miniapp.json + icon.png into dist/, and the miniapp:pack script zips dist/ into build/com.mentra.ai.local-<version>.zip. Adds an icon.png (the Mentra AI mark) and ships the packed com.mentra.ai.local-1.0.0.zip so the host preloads it like the other system miniapps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by cubic
Ports Mentra AI (local) to a backend-driven design: the agent, visual classifier, and web search now run server-side behind
@mentra/auth, while the miniapp keeps wake-word, vision, and chat with a refreshed UI. This removes AI keys from the bundle, adds model selection, optional giga‑agent delegation, app-control tools, and improves STT accuracy.New Features
Hono) with@mentra/authmiddleware and routes:POST /api/agent,POST /api/classify,POST /api/search,GET /healthz. OpenRouter + Jina keys live server-side.ask_agentdelegation to the control plane (optional), with polling and action buttons (e.g. OAuth links).start_app,stop_app,invoke_app_action(...)recorded as device actions; background executes viasession.miniapps/session.actions. Installed apps + actions are sent in context.@mentra/miniapp: managers for audio/TTS, display HUD, photos, location+weather, notifications, transcription with wake word, and query orchestration. Client talks to the backend via auth’d fetch.MENTRA_PUBLIC_*backend URL.Migration
OPENROUTER_API_KEY,JINA_API_KEY; optional delegationAGENTS_BASE_URL,MENTRA_AGENT_API_KEY. Miniapp needsMENTRA_PUBLIC_MENTRA_AI_BACKEND_URL.bun run dev(Doppler) orbun run dev:local; backend defaults to port 3131. For USB, runadb reverse tcp:3131 tcp:3131.bun run miniapp:packzipsdist/tobuild/com.mentra.ai.local-<version>.zip;build.tsstagesminiapp.jsonandicon.png.sdkVersion0.3.0,minHostVersion1.42.0. No AI keys in the bundle; ensure Doppler/infra holds backend secrets.Written for commit 52cdfa9. Summary will update on new commits.