From dc0ff1563dc18237e977c1993418c801db753cda Mon Sep 17 00:00:00 2001 From: Matthew Leibowitz Date: Fri, 22 May 2026 21:51:49 +0200 Subject: [PATCH 01/13] =?UTF-8?q?feat:=20Add=20DevFlow=20web=20inspector?= =?UTF-8?q?=20=E2=80=94=20interactive=20HTML=20mirror=20of=20running=20app?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds 'maui devflow inspector' command that serves the running MAUI app as an interactive HTML page at localhost:5223. An external tool can connect to this URL and see the native app rendered with: - Screenshot background with positioned divs for each UI element - data-* attributes exposing all DevFlow element properties - Click → tap, wheel → scroll, pointer drag → gesture proxying - Toolbar with back/refresh buttons - Auto-refresh via WebSocket when UI tree changes Architecture: CLI hosts the inspector HTTP server (not in-app), proxies all interactions to the DevFlow agent via its existing REST API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/DevFlow/inspector.md | 357 ++++++++++++ .../DevFlow/DevFlowCommands.cs | 41 ++ .../DevFlow/Inspector/HtmlRenderer.cs | 141 +++++ .../DevFlow/Inspector/InspectorServer.cs | 510 ++++++++++++++++++ .../DevFlow/Inspector/Web/devflow.js | 167 ++++++ .../Microsoft.Maui.Cli.csproj | 6 + 6 files changed, 1222 insertions(+) create mode 100644 docs/DevFlow/inspector.md create mode 100644 src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs create mode 100644 src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs create mode 100644 src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js diff --git a/docs/DevFlow/inspector.md b/docs/DevFlow/inspector.md new file mode 100644 index 00000000..004b698e --- /dev/null +++ b/docs/DevFlow/inspector.md @@ -0,0 +1,357 @@ +# DevFlow Web Inspector + +> **Status**: Design spec — V1 implementation in progress + +## Overview + +The DevFlow Web Inspector serves a running MAUI app as a fully interactive HTML page. An external inspector tool (or any browser) connects to a local URL and sees the app rendered as a live, clickable web page — complete with DOM elements matching the native visual tree. + +This enables any HTML-based inspector tool to work with a native MAUI app without custom integration. The inspector tool sees a normal website; all interaction (taps, scrolls, gestures) is transparently proxied to the real app. + +## Architecture + +``` +┌─────────────────────┐ ┌──────────────────────────┐ ┌─────────────────────┐ +│ Inspector Tool / │ HTTP │ CLI Inspector Server │ HTTP │ DevFlow Agent │ +│ Browser │ ◄─────► │ (localhost:5223) │ ◄─────► │ (device:9223) │ +│ │ │ │ │ │ +│ Sees: HTML page │ │ - Generates HTML │ │ - Visual tree API │ +│ Does: Click/scroll │ │ - Proxies API calls │ │ - Screenshot API │ +│ │ │ - WebSocket relay │ │ - Action endpoints │ +└─────────────────────┘ └──────────────────────────┘ └─────────────────────┘ +``` + +The inspector server runs on the **developer's machine** (as part of the `maui` CLI). The DevFlow agent runs **inside the native app** on any platform (device, emulator, simulator, desktop). The CLI handles agent discovery, ADB port forwarding, and all the connection plumbing. + +## Usage + +```bash +# Start the inspector server (opens at http://localhost:5223) +maui devflow inspector + +# Custom port +maui devflow inspector --port 8080 + +# Target a specific agent +maui devflow inspector --agent-port 9223 + +# Target an Android device +maui devflow inspector --device emulator-5554 +``` + +## Generated HTML Structure + +The inspector server generates an interactive HTML page with three layers: + +### Layer 1: Toolbar +```html + +``` + +### Layer 2: App Viewport with Screenshot +```html +
+ +``` + +### Layer 3: Element Divs (Transparent, Positioned) +```html +
+ +
+ +
+
+
+
+
+``` + +### Layer 4: Interaction Script +```html + +``` + +## Element Attributes + +Each `
` carries `data-*` attributes using the **exact DevFlow JSON property names** (camelCase). This gives a 1:1 mapping with the agent API — no translation needed. + +| Attribute | Source (`ElementInfo`) | Description | +|-----------|----------------------|-------------| +| `data-id` | `id` | DevFlow element ID | +| `data-parentId` | `parentId` | Parent element ID | +| `data-type` | `type` | Short type name (Button, Label, Entry) | +| `data-fullType` | `fullType` | Full .NET type (Microsoft.Maui.Controls.Button) | +| `data-framework` | `framework` | Always "maui" | +| `data-automationId` | `automationId` | AutomationId for testing | +| `data-text` | `text` | Text content | +| `data-value` | `value` | Value property | +| `data-role` | `role` | Accessibility role (button, textbox, checkbox, etc.) | +| `data-isVisible` | `isVisible` | Visibility state | +| `data-isEnabled` | `isEnabled` | Enabled state | +| `data-isFocused` | `isFocused` | Focus state | +| `data-opacity` | `opacity` | Opacity (0–1) | +| `data-traits` | `traits` | Comma-separated: interactive, focusable, scrollable, header | +| `data-gestures` | `gestures` | Comma-separated: tap, swipe, etc. | +| `data-styleClass` | `styleClass` | Comma-separated CSS style classes | +| `data-nativeType` | `nativeType` | Platform native type (e.g., Android.Widget.Button) | +| `data-nativeProperties` | `nativeProperties` | JSON-encoded native property dictionary | +| `data-frameworkProperties` | `frameworkProperties` | JSON-encoded MAUI property dictionary | + +> **Note**: HTML `data-*` attributes with camelCase suffixes work correctly. The DOM `dataset` API auto-converts them (e.g., `data-automationId` → `element.dataset.automationid`), but inspector tools read the raw attribute strings directly. + +## Agent UI Endpoints Reference + +The DevFlow agent exposes these UI endpoints. The inspector uses them as follows: + +### Read Endpoints + +| Endpoint | Method | Purpose | Inspector Use | +|----------|--------|---------|---------------| +| `/api/v1/ui/tree` | GET | Full visual tree (nested ElementInfo) | Generate HTML DOM structure | +| `/api/v1/ui/tree?depth=N` | GET | Tree limited to N levels | Optimize for deep trees | +| `/api/v1/ui/elements?type=X&text=Y&automationId=Z` | GET | Query/filter elements | Future: search | +| `/api/v1/ui/elements/{id}` | GET | Full details for one element | On-demand detail fetch | +| `/api/v1/ui/elements/{id}/properties/{name}` | GET | Read specific property | Property inspection | +| `/api/v1/ui/hit-test?x=N&y=N` | GET | Find element at coordinates | Map click to element | +| `/api/v1/ui/screenshot` | GET | PNG screenshot | Background image | + +### Action Endpoints + +| Endpoint | Method | Purpose | Inspector Use | +|----------|--------|---------|---------------| +| `/api/v1/ui/actions/tap` | POST | Tap element by ID or coordinates | Click handler | +| `/api/v1/ui/actions/scroll` | POST | Scroll by delta or to index | Wheel event handler | +| `/api/v1/ui/actions/gesture` | POST | Touch gesture (swipe, drag, pinch) | Pointer drag handler | +| `/api/v1/ui/actions/back` | POST | Navigate back | Toolbar back button | +| `/api/v1/ui/actions/fill` | POST | Fill text into Entry/Editor | Text input (V1.1) | +| `/api/v1/ui/actions/clear` | POST | Clear text from element | Text input (V1.1) | +| `/api/v1/ui/actions/key` | POST | Send key press | Key events (V1.1) | +| `/api/v1/ui/actions/focus` | POST | Focus an element | Auto on tap | +| `/api/v1/ui/actions/navigate` | POST | Shell navigation by route | URL navigation (V1.2) | +| `/api/v1/ui/actions/resize` | POST | Resize window | Not needed | +| `/api/v1/ui/actions/batch` | POST | Multiple actions at once | Optimization | + +### Mutation Endpoints + +| Endpoint | Method | Purpose | Inspector Use | +|----------|--------|---------|---------------| +| `/api/v1/ui/elements/{id}/properties/{name}` | PUT | Set property value | Live editing (V1.2) | + +### WebSocket + +| Endpoint | Purpose | Inspector Use | +|----------|---------|---------------| +| `/ws/v1/ui/events` | Real-time UI events | Auto-refresh page | + +#### Event Types + +| Event | When | Inspector Action | +|-------|------|-----------------| +| `treeChange` | After tap, fill, scroll, property set | Rebuild DOM + refresh screenshot | +| `navigation` | Shell route changed | Rebuild DOM + refresh screenshot | +| `lifecycle` | App started/stopped | Show connection status | + +Clients can subscribe to specific events: +```json +{"type": "subscribe", "data": {"events": ["treeChange", "navigation"]}} +``` + +## Interaction Model + +### Click → Tap (V1) + +```javascript +viewport.addEventListener('click', async (e) => { + const rect = viewport.getBoundingClientRect(); + const x = e.clientX - rect.left; + const y = e.clientY - rect.top; + await fetch('/api/tap', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ x, y }) + }); + await refreshScreenshot(); +}); +``` + +### Wheel → Scroll (V1) + +```javascript +viewport.addEventListener('wheel', async (e) => { + e.preventDefault(); + const rect = viewport.getBoundingClientRect(); + await fetch('/api/scroll', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + x: e.clientX - rect.left, + y: e.clientY - rect.top, + deltaX: e.deltaX, + deltaY: e.deltaY + }) + }); + await refreshScreenshot(); +}); +``` + +### Pointer Drag → Gesture (V1) + +```javascript +let gesturePoints = []; + +viewport.addEventListener('pointerdown', (e) => { + gesturePoints = [{ x: e.offsetX, y: e.offsetY, t: Date.now() }]; + viewport.setPointerCapture(e.pointerId); +}); + +viewport.addEventListener('pointermove', (e) => { + if (gesturePoints.length > 0) { + gesturePoints.push({ x: e.offsetX, y: e.offsetY, t: Date.now() }); + } +}); + +viewport.addEventListener('pointerup', async (e) => { + if (gesturePoints.length > 1) { + await fetch('/api/gesture', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ points: gesturePoints }) + }); + await refreshScreenshot(); + } + gesturePoints = []; +}); +``` + +### WebSocket Auto-Refresh (V1) + +```javascript +const ws = new WebSocket(`ws://${location.host}/ws/events`); +ws.onmessage = (e) => { + const event = JSON.parse(e.data); + if (event.type === 'treeChange' || event.type === 'navigation') { + refreshPage(); // re-fetch tree + screenshot, rebuild DOM + } +}; +ws.onclose = () => { + document.getElementById('connection-status').textContent = '○ Disconnected'; + setTimeout(connectWebSocket, 2000); +}; +``` + +### Text Input via Overlay (V1.1) + +When user taps on an element with `data-type="Entry"` or `data-role="textbox"`: +1. Show a floating `` element positioned over the element bounds +2. Pre-fill with current `data-text` value +3. On blur or Enter: send `POST /api/fill` with full text value +4. Remove overlay, refresh screenshot + +### URL-Based Navigation (V1.2) + +Shell routes map to browser URL paths: +- `http://localhost:5223/MainPage` → navigate to `//MainPage` +- `http://localhost:5223/Settings` → navigate to `//Settings` +- `http://localhost:5223/Detail?id=42` → navigate to `//Detail?id=42` + +Browser back/forward maps to app navigation. `history.pushState` keeps the URL in sync. + +## Inspector Server Routes + +| Route | Method | Description | +|-------|--------|-------------| +| `/` | GET | Generated interactive HTML page | +| `/screenshot.png` | GET | Proxied PNG from agent (cached ~200ms) | +| `/devflow.js` | GET | Embedded interaction script | +| `/api/tap` | POST | Proxy → agent `/api/v1/ui/actions/tap` | +| `/api/scroll` | POST | Proxy → agent `/api/v1/ui/actions/scroll` | +| `/api/gesture` | POST | Proxy → agent `/api/v1/ui/actions/gesture` | +| `/api/back` | POST | Proxy → agent `/api/v1/ui/actions/back` | +| `/api/fill` | POST | Proxy → agent `/api/v1/ui/actions/fill` (V1.1) | +| `/api/key` | POST | Proxy → agent `/api/v1/ui/actions/key` (V1.1) | +| `/api/tree` | GET | Proxy → agent `/api/v1/ui/tree` | +| `/ws/events` | WS | Proxy → agent `/ws/v1/ui/events` | + +## Screenshot Refresh Strategy + +After any user interaction: +1. Wait ~100ms for the app to settle (animations, layout) +2. Fetch new `/screenshot.png` +3. Swap the `` src (avoids full page rebuild for simple interactions) + +On `treeChange`/`navigation` WebSocket events: +- Full page rebuild (re-fetch tree + screenshot, regenerate DOM) + +## Versioned Roadmap + +### V1 — Interactive Mirror (Current) + +| Feature | Implementation | +|---------|---------------| +| Screenshot background | `` | +| Element divs with data-* | Nested positioned divs from tree | +| Click → tap | Coordinate-based POST | +| Scroll → scroll | Wheel event → delta POST | +| Drag → gesture | Pointer events → path POST | +| Back | Toolbar button → POST | +| Auto-refresh | WebSocket → page rebuild | +| Toolbar | Back, refresh, connection status | + +### V1.1 — Text Input + +| Feature | Implementation | +|---------|---------------| +| Text entry | Overlay `` on Entry/Editor elements | +| Key press | `keydown` → POST /api/key | +| Clear | Select-all + delete | + +### V1.2 — Navigation & Editing + +| Feature | Implementation | +|---------|---------------| +| URL = Shell route | Browser path maps to navigate endpoint | +| Deep linking | Opening URL navigates app | +| pushState | Navigation events update browser URL | +| Property editing | PUT endpoint from inspector | + +## Implementation Files + +| File | Purpose | +|------|---------| +| `src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs` | HTTP server, API proxy, WebSocket relay | +| `src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs` | Visual tree → interactive HTML generation | +| `src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js` | Client-side interaction handlers | +| `src/Cli/Microsoft.Maui.Cli/Microsoft.Maui.Cli.csproj` | EmbeddedResource for devflow.js | +| `src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs` | `inspector` subcommand registration | diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs index a42521db..95c738bb 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs @@ -1568,6 +1568,47 @@ await MauiScrollAsync(host, port, isJson, mcpCmd.SetAction(async (ctx, ct) => { await Mcp.McpServerHost.RunAsync(); }); devflowCommand.Add(mcpCmd); + // ===== Inspector command ===== + var inspectorPortOption = new Option("--port") { Description = "Inspector server port", DefaultValueFactory = _ => 5223 }; + var inspectorCmd = new Command("inspector", "Start web inspector — serves the app as an interactive HTML page"); + inspectorCmd.Add(inspectorPortOption); + inspectorCmd.SetAction(async (ctx, ct) => + { + var host = ctx.GetValue(agentHostOption)!; + var port = ctx.GetValue(agentPortOption); + var inspectorPort = ctx.GetValue(inspectorPortOption); + + var server = new Inspector.InspectorServer(inspectorPort, host, port); + server.Start(); + + var url = $"http://localhost:{inspectorPort}"; + Console.WriteLine($"DevFlow Inspector running at {url}"); + Console.WriteLine($"Proxying to agent at {host}:{port}"); + Console.WriteLine("Press Ctrl+C to stop."); + + // Try to open browser + try + { + if (OperatingSystem.IsMacOS()) + System.Diagnostics.Process.Start("open", url); + else if (OperatingSystem.IsWindows()) + System.Diagnostics.Process.Start(new System.Diagnostics.ProcessStartInfo(url) { UseShellExecute = true }); + else if (OperatingSystem.IsLinux()) + System.Diagnostics.Process.Start("xdg-open", url); + } + catch { } + + // Wait until cancelled + var tcs = new TaskCompletionSource(); + ct.Register(() => tcs.TrySetResult()); + Console.CancelKeyPress += (_, e) => { e.Cancel = true; tcs.TrySetResult(); }; + await tcs.Task; + + await server.StopAsync(); + server.Dispose(); + }); + devflowCommand.Add(inspectorCmd); + _devflowCommand = devflowCommand; return devflowCommand; diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs new file mode 100644 index 00000000..211dcd3b --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs @@ -0,0 +1,141 @@ +using System.Text; +using System.Web; +using Microsoft.Maui.DevFlow.Driver; + +namespace Microsoft.Maui.Cli.DevFlow.Inspector; + +/// +/// Generates an interactive HTML page from the DevFlow visual tree. +/// Each element becomes a positioned div with data-* attributes matching +/// the DevFlow ElementInfo property names (camelCase). +/// +public static class HtmlRenderer +{ + public static string Render(List tree, bool hasScreenshot) + { + var sb = new StringBuilder(); + sb.AppendLine(""); + sb.AppendLine(""); + sb.AppendLine(""); + sb.AppendLine(" "); + sb.AppendLine(" DevFlow Inspector"); + sb.AppendLine(" "); + sb.AppendLine(""); + sb.AppendLine(""); + + // Toolbar + sb.AppendLine(" "); + + // Determine viewport size from root element bounds + double viewportWidth = 390; + double viewportHeight = 844; + var rootBounds = tree.Count > 0 ? tree[0].Bounds : null; + if (rootBounds != null) + { + viewportWidth = rootBounds.Width; + viewportHeight = rootBounds.Height; + } + + sb.AppendLine($"
"); + + if (hasScreenshot) + { + sb.AppendLine(" \"App"); + } + + // Render element tree as nested divs + foreach (var element in tree) + { + RenderElement(sb, element, 4); + } + + sb.AppendLine("
"); + sb.AppendLine(" "); + sb.AppendLine(""); + sb.AppendLine(""); + + return sb.ToString(); + } + + private static void RenderElement(StringBuilder sb, ElementInfo element, int indent) + { + var pad = new string(' ', indent); + + // Build style for positioning + var style = new StringBuilder("position:absolute;"); + if (element.Bounds != null) + { + style.Append($"left:{element.Bounds.X:F0}px;"); + style.Append($"top:{element.Bounds.Y:F0}px;"); + style.Append($"width:{element.Bounds.Width:F0}px;"); + style.Append($"height:{element.Bounds.Height:F0}px;"); + } + + // Build data attributes + var attrs = new StringBuilder(); + attrs.Append($" data-id=\"{Escape(element.Id)}\""); + attrs.Append($" data-type=\"{Escape(element.Type)}\""); + + if (!string.IsNullOrEmpty(element.FullType)) + attrs.Append($" data-fullType=\"{Escape(element.FullType)}\""); + if (!string.IsNullOrEmpty(element.Framework)) + attrs.Append($" data-framework=\"{Escape(element.Framework)}\""); + if (!string.IsNullOrEmpty(element.AutomationId)) + attrs.Append($" data-automationId=\"{Escape(element.AutomationId)}\""); + if (!string.IsNullOrEmpty(element.Text)) + attrs.Append($" data-text=\"{Escape(element.Text)}\""); + if (!string.IsNullOrEmpty(element.Value)) + attrs.Append($" data-value=\"{Escape(element.Value)}\""); + if (!string.IsNullOrEmpty(element.Role)) + attrs.Append($" data-role=\"{Escape(element.Role)}\""); + + attrs.Append($" data-isVisible=\"{element.IsVisible.ToString().ToLowerInvariant()}\""); + attrs.Append($" data-isEnabled=\"{element.IsEnabled.ToString().ToLowerInvariant()}\""); + attrs.Append($" data-isFocused=\"{element.IsFocused.ToString().ToLowerInvariant()}\""); + attrs.Append($" data-opacity=\"{element.Opacity}\""); + + if (element.Traits is { Count: > 0 }) + attrs.Append($" data-traits=\"{Escape(string.Join(",", element.Traits))}\""); + if (element.Gestures is { Count: > 0 }) + attrs.Append($" data-gestures=\"{Escape(string.Join(",", element.Gestures))}\""); + if (element.StyleClass is { Count: > 0 }) + attrs.Append($" data-styleClass=\"{Escape(string.Join(",", element.StyleClass))}\""); + if (!string.IsNullOrEmpty(element.NativeType)) + attrs.Append($" data-nativeType=\"{Escape(element.NativeType)}\""); + + var hasChildren = element.Children is { Count: > 0 }; + + sb.Append($"{pad}
"); + + if (hasChildren) + { + sb.AppendLine(); + foreach (var child in element.Children!) + { + RenderElement(sb, child, indent + 2); + } + sb.AppendLine($"{pad}
"); + } + else + { + sb.AppendLine("
"); + } + } + + private static string Escape(string value) => HttpUtility.HtmlAttributeEncode(value); +} diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs new file mode 100644 index 00000000..c884a6c1 --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs @@ -0,0 +1,510 @@ +using System.Net; +using System.Net.Sockets; +using System.Reflection; +using System.Text; +using System.Text.Json; +using Microsoft.Maui.DevFlow.Driver; + +namespace Microsoft.Maui.Cli.DevFlow.Inspector; + +/// +/// Lightweight HTTP server that serves the DevFlow Web Inspector. +/// Generates an interactive HTML page representing the native app's visual tree +/// and proxies interaction commands to the DevFlow agent. +/// +public sealed class InspectorServer : IDisposable +{ + private TcpListener? _listener; + private CancellationTokenSource? _cts; + private Task? _listenTask; + private readonly int _port; + private readonly string _agentHost; + private readonly int _agentPort; + private byte[]? _cachedScreenshot; + private DateTime _screenshotCacheTime; + private static readonly TimeSpan ScreenshotCacheDuration = TimeSpan.FromMilliseconds(200); + + public int Port => _port; + + public InspectorServer(int port, string agentHost, int agentPort) + { + _port = port; + _agentHost = agentHost; + _agentPort = agentPort; + } + + public void Start() + { + _cts = new CancellationTokenSource(); + _listener = new TcpListener(IPAddress.Loopback, _port); + _listener.Start(); + _listenTask = AcceptLoop(_cts.Token); + } + + public async Task StopAsync() + { + _cts?.Cancel(); + _listener?.Stop(); + if (_listenTask != null) + await _listenTask.ConfigureAwait(false); + } + + public void Dispose() + { + _cts?.Cancel(); + _listener?.Stop(); + _cts?.Dispose(); + } + + private async Task AcceptLoop(CancellationToken ct) + { + while (!ct.IsCancellationRequested) + { + try + { + var client = await _listener!.AcceptTcpClientAsync(ct); + _ = HandleClientAsync(client, ct); + } + catch (OperationCanceledException) { break; } + catch (ObjectDisposedException) { break; } + catch { } + } + } + + private async Task HandleClientAsync(TcpClient client, CancellationToken ct) + { + try + { + using (client) + { + var stream = client.GetStream(); + var request = await ReadRequestAsync(stream, ct); + if (request == null) return; + + // Check for WebSocket upgrade on /ws/events + if (request.Path == "/ws/events" && + request.Headers.TryGetValue("upgrade", out var upgrade) && + upgrade.Equals("websocket", StringComparison.OrdinalIgnoreCase)) + { + await HandleWebSocketProxy(client, stream, request, ct); + return; + } + + var (statusCode, contentType, body) = await RouteAsync(request); + await WriteResponseAsync(stream, statusCode, contentType, body, ct); + } + } + catch { } + } + + private async Task<(int statusCode, string contentType, byte[] body)> RouteAsync(HttpRequestInfo request) + { + try + { + return request.Method switch + { + "GET" => request.Path switch + { + "/" or "" => await HandleRootAsync(), + "/screenshot.png" => await HandleScreenshotAsync(), + "/devflow.js" => HandleEmbeddedFile("devflow.js", "application/javascript"), + _ => (404, "text/plain", Encoding.UTF8.GetBytes("Not Found")) + }, + "POST" => request.Path switch + { + "/api/tap" => await HandleProxyTapAsync(request.Body), + "/api/scroll" => await HandleProxyScrollAsync(request.Body), + "/api/gesture" => await HandleProxyGestureAsync(request.Body), + "/api/back" => await HandleProxyBackAsync(), + "/api/fill" => await HandleProxyFillAsync(request.Body), + "/api/key" => await HandleProxyKeyAsync(request.Body), + _ => (404, "text/plain", Encoding.UTF8.GetBytes("Not Found")) + }, + _ => (405, "text/plain", Encoding.UTF8.GetBytes("Method Not Allowed")) + }; + } + catch (Exception ex) + { + return (500, "text/plain", Encoding.UTF8.GetBytes($"Error: {ex.Message}")); + } + } + + private async Task<(int, string, byte[])> HandleRootAsync() + { + using var client = new AgentClient(_agentHost, _agentPort); + var tree = await client.GetTreeAsync(); + var screenshot = await GetCachedScreenshotAsync(client); + var html = HtmlRenderer.Render(tree, screenshot?.Length > 0); + return (200, "text/html; charset=utf-8", Encoding.UTF8.GetBytes(html)); + } + + private async Task<(int, string, byte[])> HandleScreenshotAsync() + { + using var client = new AgentClient(_agentHost, _agentPort); + var png = await GetCachedScreenshotAsync(client); + if (png == null || png.Length == 0) + return (404, "text/plain", Encoding.UTF8.GetBytes("No screenshot available")); + return (200, "image/png", png); + } + + private (int, string, byte[]) HandleEmbeddedFile(string fileName, string contentType) + { + var assembly = Assembly.GetExecutingAssembly(); + var resourceName = $"Microsoft.Maui.Cli.DevFlow.Inspector.Web.{fileName}"; + using var stream = assembly.GetManifestResourceStream(resourceName); + if (stream == null) + return (404, "text/plain", Encoding.UTF8.GetBytes($"Resource not found: {resourceName}")); + + using var ms = new MemoryStream(); + stream.CopyTo(ms); + return (200, contentType, ms.ToArray()); + } + + private async Task GetCachedScreenshotAsync(AgentClient client) + { + if (_cachedScreenshot != null && DateTime.UtcNow - _screenshotCacheTime < ScreenshotCacheDuration) + return _cachedScreenshot; + + _cachedScreenshot = await client.ScreenshotAsync(); + _screenshotCacheTime = DateTime.UtcNow; + return _cachedScreenshot; + } + + // ── Proxy handlers ── + + private async Task<(int, string, byte[])> HandleProxyTapAsync(string? body) + { + if (string.IsNullOrEmpty(body)) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"Body required\"}")); + + using var doc = JsonDocument.Parse(body); + var root = doc.RootElement; + + // Support coordinate-based tap: hit-test first, then tap the element + if (root.TryGetProperty("x", out var xProp) && root.TryGetProperty("y", out var yProp)) + { + var x = xProp.GetDouble(); + var y = yProp.GetDouble(); + + using var client = new AgentClient(_agentHost, _agentPort); + var hitResult = await client.HitTestAsync(x, y); + + // Parse hit-test result to get element ID + using var hitDoc = JsonDocument.Parse(hitResult); + if (hitDoc.RootElement.TryGetProperty("id", out var idProp)) + { + var elementId = idProp.GetString(); + if (!string.IsNullOrEmpty(elementId)) + { + var success = await client.TapAsync(elementId); + _cachedScreenshot = null; // invalidate cache + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + } + return (404, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"No element at coordinates\"}")); + } + + // Support elementId-based tap + if (root.TryGetProperty("elementId", out var elIdProp)) + { + var elementId = elIdProp.GetString(); + if (!string.IsNullOrEmpty(elementId)) + { + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.TapAsync(elementId); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + } + + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"x/y or elementId required\"}")); + } + + private async Task<(int, string, byte[])> HandleProxyScrollAsync(string? body) + { + if (string.IsNullOrEmpty(body)) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"Body required\"}")); + + using var doc = JsonDocument.Parse(body); + var root = doc.RootElement; + + var deltaX = root.TryGetProperty("deltaX", out var dxProp) ? dxProp.GetDouble() : 0; + var deltaY = root.TryGetProperty("deltaY", out var dyProp) ? dyProp.GetDouble() : 0; + string? elementId = null; + + // If coordinates provided, hit-test to find the scrollable element + if (root.TryGetProperty("x", out var xProp) && root.TryGetProperty("y", out var yProp)) + { + using var hitClient = new AgentClient(_agentHost, _agentPort); + var hitResult = await hitClient.HitTestAsync(xProp.GetDouble(), yProp.GetDouble()); + using var hitDoc = JsonDocument.Parse(hitResult); + if (hitDoc.RootElement.TryGetProperty("id", out var idProp)) + elementId = idProp.GetString(); + } + + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.ScrollAsync(elementId: elementId, deltaX: deltaX, deltaY: deltaY); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + + private async Task<(int, string, byte[])> HandleProxyGestureAsync(string? body) + { + if (string.IsNullOrEmpty(body)) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"Body required\"}")); + + using var doc = JsonDocument.Parse(body); + var root = doc.RootElement; + + // Determine swipe direction from gesture points + if (root.TryGetProperty("points", out var pointsArr) && pointsArr.GetArrayLength() >= 2) + { + var first = pointsArr[0]; + var last = pointsArr[pointsArr.GetArrayLength() - 1]; + var dx = last.GetProperty("x").GetDouble() - first.GetProperty("x").GetDouble(); + var dy = last.GetProperty("y").GetDouble() - first.GetProperty("y").GetDouble(); + + var direction = Math.Abs(dx) > Math.Abs(dy) + ? (dx > 0 ? "right" : "left") + : (dy > 0 ? "down" : "up"); + + var distance = Math.Sqrt(dx * dx + dy * dy); + + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.GestureAsync("swipe", direction: direction, distance: distance); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"points array required\"}")); + } + + private async Task<(int, string, byte[])> HandleProxyBackAsync() + { + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.BackAsync(); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + + private async Task<(int, string, byte[])> HandleProxyFillAsync(string? body) + { + if (string.IsNullOrEmpty(body)) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"Body required\"}")); + + using var doc = JsonDocument.Parse(body); + var root = doc.RootElement; + + var elementId = root.TryGetProperty("elementId", out var idProp) ? idProp.GetString() : null; + var text = root.TryGetProperty("text", out var textProp) ? textProp.GetString() : null; + + if (string.IsNullOrEmpty(elementId) || text == null) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"elementId and text required\"}")); + + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.FillAsync(elementId, text); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + + private async Task<(int, string, byte[])> HandleProxyKeyAsync(string? body) + { + if (string.IsNullOrEmpty(body)) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"Body required\"}")); + + using var doc = JsonDocument.Parse(body); + var root = doc.RootElement; + + var key = root.TryGetProperty("key", out var keyProp) ? keyProp.GetString() : null; + var elementId = root.TryGetProperty("elementId", out var idProp) ? idProp.GetString() : null; + + if (string.IsNullOrEmpty(key)) + return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"key required\"}")); + + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.KeyAsync(key, elementId); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } + + // ── WebSocket proxy (pass-through to agent /ws/v1/ui/events) ── + + private async Task HandleWebSocketProxy(TcpClient tcpClient, NetworkStream clientStream, HttpRequestInfo request, CancellationToken ct) + { + // Complete WebSocket handshake with browser + if (!request.Headers.TryGetValue("sec-websocket-key", out var wsKey)) + return; + + var acceptKey = Convert.ToBase64String( + System.Security.Cryptography.SHA1.HashData( + Encoding.UTF8.GetBytes(wsKey + "258EAFA5-E914-47DA-95CA-5AB5DC4B46D6"))); + + var handshake = $"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {acceptKey}\r\n\r\n"; + await clientStream.WriteAsync(Encoding.UTF8.GetBytes(handshake), ct); + await clientStream.FlushAsync(ct); + + // Connect to agent WebSocket and relay messages + using var agentWs = new System.Net.WebSockets.ClientWebSocket(); + try + { + await agentWs.ConnectAsync(new Uri($"ws://{_agentHost}:{_agentPort}/ws/v1/ui/events"), ct); + + // Subscribe to all events + var subscribe = Encoding.UTF8.GetBytes("{\"type\":\"subscribe\",\"data\":{\"events\":[\"all\"]}}"); + await agentWs.SendAsync(subscribe, System.Net.WebSockets.WebSocketMessageType.Text, true, ct); + + // Relay agent messages to browser + var buffer = new byte[8192]; + while (!ct.IsCancellationRequested && agentWs.State == System.Net.WebSockets.WebSocketState.Open) + { + var result = await agentWs.ReceiveAsync(buffer, ct); + if (result.MessageType == System.Net.WebSockets.WebSocketMessageType.Close) + break; + + if (result.MessageType == System.Net.WebSockets.WebSocketMessageType.Text) + { + var payload = buffer.AsMemory(0, result.Count).ToArray(); + await SendWebSocketFrameAsync(clientStream, payload, ct); + } + } + } + catch { } + } + + private static async Task SendWebSocketFrameAsync(NetworkStream stream, byte[] payload, CancellationToken ct) + { + // Build a text frame (FIN + opcode 0x1) + var frame = new List { 0x81 }; // FIN + text + if (payload.Length < 126) + frame.Add((byte)payload.Length); + else if (payload.Length <= 65535) + { + frame.Add(126); + frame.Add((byte)(payload.Length >> 8)); + frame.Add((byte)(payload.Length & 0xFF)); + } + else + { + frame.Add(127); + var len = (long)payload.Length; + for (int i = 7; i >= 0; i--) + frame.Add((byte)((len >> (i * 8)) & 0xFF)); + } + frame.AddRange(payload); + await stream.WriteAsync(frame.ToArray(), ct); + await stream.FlushAsync(ct); + } + + // ── HTTP parsing helpers ── + + private static async Task ReadRequestAsync(NetworkStream stream, CancellationToken ct) + { + var buffer = new byte[8192]; + int read; + using var timeoutCts = CancellationTokenSource.CreateLinkedTokenSource(ct); + timeoutCts.CancelAfter(TimeSpan.FromSeconds(5)); + + try + { + read = await stream.ReadAsync(buffer, timeoutCts.Token); + if (read == 0) return null; + } + catch { return null; } + + var raw = Encoding.UTF8.GetString(buffer, 0, read); + var headerEnd = raw.IndexOf("\r\n\r\n", StringComparison.Ordinal); + if (headerEnd < 0) return null; + + var headerSection = raw[..headerEnd]; + var bodySection = raw[(headerEnd + 4)..]; + + var lines = headerSection.Split("\r\n"); + if (lines.Length == 0) return null; + + var requestLine = lines[0].Split(' '); + if (requestLine.Length < 2) return null; + + var method = requestLine[0].ToUpperInvariant(); + var path = requestLine[1].Split('?')[0].TrimEnd('/'); + if (string.IsNullOrEmpty(path)) path = "/"; + + var headers = new Dictionary(StringComparer.OrdinalIgnoreCase); + for (int i = 1; i < lines.Length; i++) + { + var colonIdx = lines[i].IndexOf(':'); + if (colonIdx > 0) + { + var key = lines[i][..colonIdx].Trim(); + var value = lines[i][(colonIdx + 1)..].Trim(); + headers[key] = value; + } + } + + // Read remaining body if Content-Length indicates more + var body = bodySection; + if (headers.TryGetValue("content-length", out var clStr) && int.TryParse(clStr, out var contentLength)) + { + while (Encoding.UTF8.GetByteCount(body) < contentLength) + { + var extraRead = await stream.ReadAsync(buffer, ct); + if (extraRead == 0) break; + body += Encoding.UTF8.GetString(buffer, 0, extraRead); + } + } + + return new HttpRequestInfo + { + Method = method, + Path = path, + Headers = headers, + Body = body + }; + } + + private static async Task WriteResponseAsync(NetworkStream stream, int statusCode, string contentType, byte[] body, CancellationToken ct) + { + var statusText = statusCode switch + { + 200 => "OK", + 400 => "Bad Request", + 404 => "Not Found", + 405 => "Method Not Allowed", + 500 => "Internal Server Error", + _ => "Unknown" + }; + + var header = $"HTTP/1.1 {statusCode} {statusText}\r\n" + + $"Content-Type: {contentType}\r\n" + + $"Content-Length: {body.Length}\r\n" + + "Access-Control-Allow-Origin: *\r\n" + + "Access-Control-Allow-Methods: GET, POST, OPTIONS\r\n" + + "Access-Control-Allow-Headers: Content-Type\r\n" + + "Connection: close\r\n\r\n"; + + await stream.WriteAsync(Encoding.UTF8.GetBytes(header), ct); + await stream.WriteAsync(body, ct); + await stream.FlushAsync(ct); + } + + private sealed class HttpRequestInfo + { + public string Method { get; init; } = ""; + public string Path { get; init; } = ""; + public Dictionary Headers { get; init; } = new(); + public string? Body { get; init; } + } +} diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js new file mode 100644 index 00000000..34333797 --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -0,0 +1,167 @@ +// DevFlow Web Inspector — Interaction Script +// Intercepts browser events and proxies them to the native app via the inspector server. +(function () { + 'use strict'; + + const viewport = document.getElementById('app-viewport'); + const screenshot = document.getElementById('screenshot'); + const btnBack = document.getElementById('btn-back'); + const btnRefresh = document.getElementById('btn-refresh'); + const statusEl = document.getElementById('connection-status'); + + let gesturePoints = []; + let isGesturing = false; + let isDragging = false; + + // ── Click → Tap ── + viewport.addEventListener('click', async (e) => { + if (isDragging) return; // don't tap if we just finished a drag + if (e.target.closest('#devflow-toolbar')) return; + + const rect = viewport.getBoundingClientRect(); + const x = e.clientX - rect.left; + const y = e.clientY - rect.top; + + try { + await fetch('/api/tap', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ x, y }) + }); + await refreshScreenshot(); + } catch (err) { + console.error('Tap failed:', err); + } + }); + + // ── Wheel → Scroll ── + viewport.addEventListener('wheel', async (e) => { + e.preventDefault(); + const rect = viewport.getBoundingClientRect(); + const x = e.clientX - rect.left; + const y = e.clientY - rect.top; + + try { + await fetch('/api/scroll', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ x, y, deltaX: e.deltaX, deltaY: e.deltaY }) + }); + await refreshScreenshot(); + } catch (err) { + console.error('Scroll failed:', err); + } + }, { passive: false }); + + // ── Pointer Drag → Gesture ── + viewport.addEventListener('pointerdown', (e) => { + if (e.target.closest('#devflow-toolbar')) return; + gesturePoints = [{ x: e.offsetX, y: e.offsetY, t: Date.now() }]; + isGesturing = true; + isDragging = false; + viewport.setPointerCapture(e.pointerId); + }); + + viewport.addEventListener('pointermove', (e) => { + if (!isGesturing) return; + gesturePoints.push({ x: e.offsetX, y: e.offsetY, t: Date.now() }); + if (gesturePoints.length > 3) isDragging = true; + }); + + viewport.addEventListener('pointerup', async (e) => { + if (!isGesturing) return; + isGesturing = false; + + // Only send gesture if there was meaningful movement (> 20px) + if (gesturePoints.length >= 2) { + const first = gesturePoints[0]; + const last = gesturePoints[gesturePoints.length - 1]; + const dist = Math.sqrt(Math.pow(last.x - first.x, 2) + Math.pow(last.y - first.y, 2)); + + if (dist > 20) { + try { + await fetch('/api/gesture', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ points: gesturePoints }) + }); + await refreshScreenshot(); + } catch (err) { + console.error('Gesture failed:', err); + } + } + } + + gesturePoints = []; + // Reset isDragging after a short delay so the click handler can check it + setTimeout(() => { isDragging = false; }, 50); + }); + + // ── Toolbar: Back ── + btnBack.addEventListener('click', async (e) => { + e.stopPropagation(); + try { + await fetch('/api/back', { method: 'POST' }); + await refreshPage(); + } catch (err) { + console.error('Back failed:', err); + } + }); + + // ── Toolbar: Refresh ── + btnRefresh.addEventListener('click', async (e) => { + e.stopPropagation(); + await refreshPage(); + }); + + // ── Screenshot refresh ── + async function refreshScreenshot() { + // Wait for app to settle + await sleep(100); + if (screenshot) { + screenshot.src = '/screenshot.png?t=' + Date.now(); + } + } + + async function refreshPage() { + location.reload(); + } + + function sleep(ms) { + return new Promise(resolve => setTimeout(resolve, ms)); + } + + // ── WebSocket for live updates ── + function connectWebSocket() { + const protocol = location.protocol === 'https:' ? 'wss:' : 'ws:'; + const ws = new WebSocket(`${protocol}//${location.host}/ws/events`); + + ws.onopen = () => { + statusEl.textContent = '● Connected'; + statusEl.style.color = '#4ec9b0'; + }; + + ws.onmessage = (e) => { + try { + const event = JSON.parse(e.data); + if (event.type === 'treeChange' || event.type === 'navigation') { + // Debounce rapid updates + clearTimeout(ws._refreshTimer); + ws._refreshTimer = setTimeout(() => refreshPage(), 200); + } + } catch { } + }; + + ws.onclose = () => { + statusEl.textContent = '○ Disconnected'; + statusEl.style.color = '#f44747'; + setTimeout(connectWebSocket, 2000); + }; + + ws.onerror = () => { + ws.close(); + }; + } + + connectWebSocket(); +})(); diff --git a/src/Cli/Microsoft.Maui.Cli/Microsoft.Maui.Cli.csproj b/src/Cli/Microsoft.Maui.Cli/Microsoft.Maui.Cli.csproj index c5eeb9ad..9e59cc8e 100644 --- a/src/Cli/Microsoft.Maui.Cli/Microsoft.Maui.Cli.csproj +++ b/src/Cli/Microsoft.Maui.Cli/Microsoft.Maui.Cli.csproj @@ -83,4 +83,10 @@ + + + false + + + From 530cb2be9516aeca16b5e2f137ecc6044fe77cee Mon Sep 17 00:00:00 2001 From: Matthew Leibowitz Date: Fri, 22 May 2026 22:11:07 +0200 Subject: [PATCH 02/13] refactor: Strip inspector chrome, use template + separate CSS, fix viewport sizing - Remove toolbar and hover highlighting (host inspector provides its own) - Extract HTML to inspector.html template with {{placeholders}} - Separate CSS into devflow.css (editable with proper syntax highlighting) - Use actual screenshot PNG dimensions for viewport (not tree bounds) - Read PNG IHDR chunk for width/height - Add Playwright tests validating HTML structure, no chrome, correct sizing - Add ValidateXcodeVersion=false to Directory.Build.props Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Directory.Build.props | 2 + .../DevFlow/Inspector/HtmlRenderer.cs | 93 ++++++------ .../DevFlow/Inspector/InspectorServer.cs | 22 ++- .../DevFlow/Inspector/Web/devflow.css | 31 ++++ .../DevFlow/Inspector/Web/devflow.js | 41 +----- .../DevFlow/Inspector/Web/inspector.html | 16 +++ tests/Inspector.Playwright/inspector.spec.js | 133 ++++++++++++++++++ tests/Inspector.Playwright/package-lock.json | 79 +++++++++++ tests/Inspector.Playwright/package.json | 16 +++ .../Inspector.Playwright/playwright.config.js | 10 ++ 10 files changed, 355 insertions(+), 88 deletions(-) create mode 100644 src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css create mode 100644 src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html create mode 100644 tests/Inspector.Playwright/inspector.spec.js create mode 100644 tests/Inspector.Playwright/package-lock.json create mode 100644 tests/Inspector.Playwright/package.json create mode 100644 tests/Inspector.Playwright/playwright.config.js diff --git a/Directory.Build.props b/Directory.Build.props index 994a2a14..b44030d5 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -22,6 +22,8 @@ false true + + false diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs index 211dcd3b..479d01ff 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs @@ -1,3 +1,4 @@ +using System.Reflection; using System.Text; using System.Web; using Microsoft.Maui.DevFlow.Driver; @@ -6,70 +7,66 @@ namespace Microsoft.Maui.Cli.DevFlow.Inspector; /// /// Generates an interactive HTML page from the DevFlow visual tree. +/// Uses inspector.html as a template and injects the element tree. /// Each element becomes a positioned div with data-* attributes matching /// the DevFlow ElementInfo property names (camelCase). /// public static class HtmlRenderer { - public static string Render(List tree, bool hasScreenshot) + private static string? _templateCache; + + public static string Render(List tree, bool hasScreenshot, int screenshotWidth = 0, int screenshotHeight = 0) { - var sb = new StringBuilder(); - sb.AppendLine(""); - sb.AppendLine(""); - sb.AppendLine(""); - sb.AppendLine(" "); - sb.AppendLine(" DevFlow Inspector"); - sb.AppendLine(" "); - sb.AppendLine(""); - sb.AppendLine(""); - - // Toolbar - sb.AppendLine(" "); - - // Determine viewport size from root element bounds - double viewportWidth = 390; - double viewportHeight = 844; - var rootBounds = tree.Count > 0 ? tree[0].Bounds : null; - if (rootBounds != null) + var template = GetTemplate(); + + // Use screenshot dimensions as viewport size (most reliable), + // fall back to root element bounds, then default + double viewportWidth, viewportHeight; + if (screenshotWidth > 0 && screenshotHeight > 0) { - viewportWidth = rootBounds.Width; - viewportHeight = rootBounds.Height; + viewportWidth = screenshotWidth; + viewportHeight = screenshotHeight; } - - sb.AppendLine($"
"); - - if (hasScreenshot) + else { - sb.AppendLine(" \"App"); + var rootBounds = tree.Count > 0 ? tree[0].Bounds : null; + viewportWidth = rootBounds is { Width: > 0 } ? rootBounds.Width : 800; + viewportHeight = rootBounds is { Height: > 0 } ? rootBounds.Height : 600; } - // Render element tree as nested divs + // Build the elements HTML + var elements = new StringBuilder(); foreach (var element in tree) { - RenderElement(sb, element, 4); + RenderElement(elements, element, 4); } - sb.AppendLine("
"); - sb.AppendLine(" "); - sb.AppendLine(""); - sb.AppendLine(""); + // Build screenshot tag + var screenshotHtml = hasScreenshot + ? "\"App" + : ""; - return sb.ToString(); + // Replace template placeholders + var html = template + .Replace("{{VIEWPORT_WIDTH}}", viewportWidth.ToString("F0")) + .Replace("{{VIEWPORT_HEIGHT}}", viewportHeight.ToString("F0")) + .Replace("{{SCREENSHOT}}", screenshotHtml) + .Replace("{{ELEMENTS}}", elements.ToString()); + + return html; + } + + private static string GetTemplate() + { + if (_templateCache != null) return _templateCache; + + var assembly = Assembly.GetExecutingAssembly(); + var resourceName = "Microsoft.Maui.Cli.DevFlow.Inspector.Web.inspector.html"; + using var stream = assembly.GetManifestResourceStream(resourceName) + ?? throw new InvalidOperationException($"Embedded resource not found: {resourceName}"); + using var reader = new StreamReader(stream); + _templateCache = reader.ReadToEnd(); + return _templateCache; } private static void RenderElement(StringBuilder sb, ElementInfo element, int indent) diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs index c884a6c1..253de4ba 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs @@ -108,6 +108,7 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) "/" or "" => await HandleRootAsync(), "/screenshot.png" => await HandleScreenshotAsync(), "/devflow.js" => HandleEmbeddedFile("devflow.js", "application/javascript"), + "/devflow.css" => HandleEmbeddedFile("devflow.css", "text/css"), _ => (404, "text/plain", Encoding.UTF8.GetBytes("Not Found")) }, "POST" => request.Path switch @@ -134,10 +135,29 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) using var client = new AgentClient(_agentHost, _agentPort); var tree = await client.GetTreeAsync(); var screenshot = await GetCachedScreenshotAsync(client); - var html = HtmlRenderer.Render(tree, screenshot?.Length > 0); + var hasScreenshot = screenshot?.Length > 0; + + // Get screenshot dimensions for accurate viewport sizing + int width = 0, height = 0; + if (hasScreenshot) + { + (width, height) = GetPngDimensions(screenshot!); + } + + var html = HtmlRenderer.Render(tree, hasScreenshot, width, height); return (200, "text/html; charset=utf-8", Encoding.UTF8.GetBytes(html)); } + /// Reads width/height from PNG IHDR chunk (bytes 16-23). + private static (int width, int height) GetPngDimensions(byte[] png) + { + if (png.Length < 24) return (0, 0); + // PNG IHDR: width at offset 16 (4 bytes big-endian), height at offset 20 + int w = (png[16] << 24) | (png[17] << 16) | (png[18] << 8) | png[19]; + int h = (png[20] << 24) | (png[21] << 16) | (png[22] << 8) | png[23]; + return (w, h); + } + private async Task<(int, string, byte[])> HandleScreenshotAsync() { using var client = new AgentClient(_agentHost, _agentPort); diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css new file mode 100644 index 00000000..2fda1d33 --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css @@ -0,0 +1,31 @@ +* { + margin: 0; + padding: 0; + box-sizing: border-box; +} + +body { + font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; + background: #1e1e1e; + color: #fff; +} + +#app-viewport { + position: relative; + overflow: hidden; +} + +#screenshot { + position: absolute; + top: 0; + left: 0; + width: 100%; + height: 100%; + pointer-events: none; + user-select: none; +} + +.devflow-element { + position: absolute; + box-sizing: border-box; +} diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js index 34333797..39edd08d 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -5,9 +5,6 @@ const viewport = document.getElementById('app-viewport'); const screenshot = document.getElementById('screenshot'); - const btnBack = document.getElementById('btn-back'); - const btnRefresh = document.getElementById('btn-refresh'); - const statusEl = document.getElementById('connection-status'); let gesturePoints = []; let isGesturing = false; @@ -15,8 +12,7 @@ // ── Click → Tap ── viewport.addEventListener('click', async (e) => { - if (isDragging) return; // don't tap if we just finished a drag - if (e.target.closest('#devflow-toolbar')) return; + if (isDragging) return; const rect = viewport.getBoundingClientRect(); const x = e.clientX - rect.left; @@ -55,7 +51,6 @@ // ── Pointer Drag → Gesture ── viewport.addEventListener('pointerdown', (e) => { - if (e.target.closest('#devflow-toolbar')) return; gesturePoints = [{ x: e.offsetX, y: e.offsetY, t: Date.now() }]; isGesturing = true; isDragging = false; @@ -72,7 +67,6 @@ if (!isGesturing) return; isGesturing = false; - // Only send gesture if there was meaningful movement (> 20px) if (gesturePoints.length >= 2) { const first = gesturePoints[0]; const last = gesturePoints[gesturePoints.length - 1]; @@ -93,40 +87,17 @@ } gesturePoints = []; - // Reset isDragging after a short delay so the click handler can check it setTimeout(() => { isDragging = false; }, 50); }); - // ── Toolbar: Back ── - btnBack.addEventListener('click', async (e) => { - e.stopPropagation(); - try { - await fetch('/api/back', { method: 'POST' }); - await refreshPage(); - } catch (err) { - console.error('Back failed:', err); - } - }); - - // ── Toolbar: Refresh ── - btnRefresh.addEventListener('click', async (e) => { - e.stopPropagation(); - await refreshPage(); - }); - // ── Screenshot refresh ── async function refreshScreenshot() { - // Wait for app to settle await sleep(100); if (screenshot) { screenshot.src = '/screenshot.png?t=' + Date.now(); } } - async function refreshPage() { - location.reload(); - } - function sleep(ms) { return new Promise(resolve => setTimeout(resolve, ms)); } @@ -136,25 +107,17 @@ const protocol = location.protocol === 'https:' ? 'wss:' : 'ws:'; const ws = new WebSocket(`${protocol}//${location.host}/ws/events`); - ws.onopen = () => { - statusEl.textContent = '● Connected'; - statusEl.style.color = '#4ec9b0'; - }; - ws.onmessage = (e) => { try { const event = JSON.parse(e.data); if (event.type === 'treeChange' || event.type === 'navigation') { - // Debounce rapid updates clearTimeout(ws._refreshTimer); - ws._refreshTimer = setTimeout(() => refreshPage(), 200); + ws._refreshTimer = setTimeout(() => location.reload(), 200); } } catch { } }; ws.onclose = () => { - statusEl.textContent = '○ Disconnected'; - statusEl.style.color = '#f44747'; setTimeout(connectWebSocket, 2000); }; diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html new file mode 100644 index 00000000..37b6c02f --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html @@ -0,0 +1,16 @@ + + + + + DevFlow Inspector + + + +
+ {{SCREENSHOT}} + {{ELEMENTS}} +
+ + + + diff --git a/tests/Inspector.Playwright/inspector.spec.js b/tests/Inspector.Playwright/inspector.spec.js new file mode 100644 index 00000000..cfae267f --- /dev/null +++ b/tests/Inspector.Playwright/inspector.spec.js @@ -0,0 +1,133 @@ +// @ts-check +const { test, expect } = require('@playwright/test'); + +test.describe('DevFlow Inspector HTML output', () => { + + test('page loads with correct viewport dimensions from screenshot', async ({ page }) => { + await page.goto('/'); + + const viewport = page.locator('#app-viewport'); + await expect(viewport).toBeVisible(); + + // Viewport should use actual screenshot dimensions (not hardcoded 390x844) + const style = await viewport.getAttribute('style'); + expect(style).toContain('width:'); + expect(style).toContain('height:'); + + // Should NOT be iPhone defaults + expect(style).not.toContain('width:390px'); + expect(style).not.toContain('height:844px'); + }); + + test('page contains screenshot image', async ({ page }) => { + await page.goto('/'); + + const screenshot = page.locator('#screenshot'); + await expect(screenshot).toBeVisible(); + expect(await screenshot.getAttribute('src')).toBe('/screenshot.png'); + }); + + test('page has no toolbar or inspector chrome', async ({ page }) => { + await page.goto('/'); + + // Should NOT have toolbar elements — the host inspector adds its own + await expect(page.locator('#devflow-toolbar')).toHaveCount(0); + await expect(page.locator('#btn-back')).toHaveCount(0); + await expect(page.locator('#btn-refresh')).toHaveCount(0); + await expect(page.locator('#connection-status')).toHaveCount(0); + }); + + test('elements have no hover highlighting styles', async ({ page }) => { + await page.goto('/'); + + // Check that CSS doesn't include hover outline + const cssResponse = await page.request.get('/devflow.css'); + const cssText = await cssResponse.text(); + expect(cssText).not.toContain(':hover'); + expect(cssText).not.toContain('outline'); + }); + + test('elements rendered as positioned divs with data attributes', async ({ page }) => { + await page.goto('/'); + + const elements = page.locator('.devflow-element'); + const count = await elements.count(); + expect(count).toBeGreaterThan(0); + + // First element should have required data attributes + const first = elements.first(); + expect(await first.getAttribute('data-id')).toBeTruthy(); + expect(await first.getAttribute('data-type')).toBeTruthy(); + }); + + test('elements have correct positioning styles', async ({ page }) => { + await page.goto('/'); + + // Find an element with bounds that has positive dimensions + const elements = page.locator('.devflow-element[style*="left:"]'); + const count = await elements.count(); + expect(count).toBeGreaterThan(0); + + const style = await elements.first().getAttribute('style'); + expect(style).toContain('position:absolute'); + expect(style).toMatch(/left:\d/); + expect(style).toMatch(/top:\d/); + }); + + test('screenshot endpoint returns PNG', async ({ page }) => { + const response = await page.request.get('/screenshot.png'); + expect(response.status()).toBe(200); + expect(response.headers()['content-type']).toBe('image/png'); + + const body = await response.body(); + // PNG magic bytes + expect(body[0]).toBe(0x89); + expect(body[1]).toBe(0x50); // P + expect(body[2]).toBe(0x4E); // N + expect(body[3]).toBe(0x47); // G + }); + + test('CSS served as separate file', async ({ page }) => { + const response = await page.request.get('/devflow.css'); + expect(response.status()).toBe(200); + expect(response.headers()['content-type']).toBe('text/css'); + + const text = await response.text(); + expect(text).toContain('#app-viewport'); + expect(text).toContain('.devflow-element'); + }); + + test('JS served as separate file', async ({ page }) => { + const response = await page.request.get('/devflow.js'); + expect(response.status()).toBe(200); + expect(response.headers()['content-type']).toBe('application/javascript'); + + const text = await response.text(); + expect(text).toContain('app-viewport'); + expect(text).toContain('/api/tap'); + }); + + test('element tree is nested (children inside parents)', async ({ page }) => { + await page.goto('/'); + + // Find a parent element that contains child elements + const nestedParent = page.locator('.devflow-element > .devflow-element'); + const count = await nestedParent.count(); + expect(count).toBeGreaterThan(0); + }); + + test('data attributes use camelCase naming from DevFlow API', async ({ page }) => { + await page.goto('/'); + + // Check attributes directly on elements rather than raw HTML + const elemWithVis = page.locator('.devflow-element[data-isVisible]'); + expect(await elemWithVis.count()).toBeGreaterThan(0); + + const elemWithEnabled = page.locator('.devflow-element[data-isEnabled]'); + expect(await elemWithEnabled.count()).toBeGreaterThan(0); + + // Verify camelCase naming convention + const elemWithFullType = page.locator('.devflow-element[data-fullType]'); + expect(await elemWithFullType.count()).toBeGreaterThan(0); + }); +}); diff --git a/tests/Inspector.Playwright/package-lock.json b/tests/Inspector.Playwright/package-lock.json new file mode 100644 index 00000000..9b153d18 --- /dev/null +++ b/tests/Inspector.Playwright/package-lock.json @@ -0,0 +1,79 @@ +{ + "name": "inspector.playwright", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "inspector.playwright", + "version": "1.0.0", + "license": "ISC", + "devDependencies": { + "@playwright/test": "^1.60.0" + } + }, + "node_modules/@playwright/test": { + "version": "1.60.0", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.60.0.tgz", + "integrity": "sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright": "1.60.0" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/playwright": { + "version": "1.60.0", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.60.0.tgz", + "integrity": "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright-core": "1.60.0" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=18" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/playwright-core": { + "version": "1.60.0", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.60.0.tgz", + "integrity": "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=18" + } + } + } +} diff --git a/tests/Inspector.Playwright/package.json b/tests/Inspector.Playwright/package.json new file mode 100644 index 00000000..b1553e12 --- /dev/null +++ b/tests/Inspector.Playwright/package.json @@ -0,0 +1,16 @@ +{ + "name": "inspector.playwright", + "version": "1.0.0", + "description": "", + "main": "index.js", + "scripts": { + "test": "echo \"Error: no test specified\" && exit 1" + }, + "keywords": [], + "author": "", + "license": "ISC", + "type": "commonjs", + "devDependencies": { + "@playwright/test": "^1.60.0" + } +} diff --git a/tests/Inspector.Playwright/playwright.config.js b/tests/Inspector.Playwright/playwright.config.js new file mode 100644 index 00000000..e7709523 --- /dev/null +++ b/tests/Inspector.Playwright/playwright.config.js @@ -0,0 +1,10 @@ +// @ts-check +const { defineConfig } = require('@playwright/test'); + +module.exports = defineConfig({ + testDir: '.', + timeout: 30000, + use: { + baseURL: 'http://localhost:5223', + }, +}); From 53cf03129e744752d2b7161ee25b90644bb4e684 Mon Sep 17 00:00:00 2001 From: Matthew Leibowitz Date: Fri, 22 May 2026 22:25:55 +0200 Subject: [PATCH 03/13] fix: Use agent status API for viewport size, add zoom-to-fit, C# Playwright tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Viewport dimensions now come from agent status API (device.windowWidth/Height) which works cross-platform (Android, iOS, Mac, Windows) - Fallback to PNG IHDR dimensions if status unavailable - CSS zoom-to-fit: viewport scales down to fit browser window (never upscales) - JS coordinate conversion accounts for zoom scale factor - Replace Node.js Playwright tests with proper C# xUnit + Microsoft.Playwright tests in src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/ - 12 tests covering: viewport sizing, zoom, no chrome, element positioning, nested tree, data attributes, click→tap interaction, screenshot endpoint Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Directory.Packages.props | 1 + .../DevFlow/Inspector/InspectorServer.cs | 20 +- .../DevFlow/Inspector/Web/devflow.css | 6 + .../DevFlow/Inspector/Web/devflow.js | 40 ++- .../DevFlow/Inspector/Web/inspector.html | 2 +- .../InspectorPageTests.cs | 257 ++++++++++++++++++ ...rosoft.Maui.DevFlow.Inspector.Tests.csproj | 23 ++ tests/Inspector.Playwright/inspector.spec.js | 133 --------- tests/Inspector.Playwright/package-lock.json | 79 ------ tests/Inspector.Playwright/package.json | 16 -- .../Inspector.Playwright/playwright.config.js | 10 - 11 files changed, 334 insertions(+), 253 deletions(-) create mode 100644 src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs create mode 100644 src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/Microsoft.Maui.DevFlow.Inspector.Tests.csproj delete mode 100644 tests/Inspector.Playwright/inspector.spec.js delete mode 100644 tests/Inspector.Playwright/package-lock.json delete mode 100644 tests/Inspector.Playwright/package.json delete mode 100644 tests/Inspector.Playwright/playwright.config.js diff --git a/Directory.Packages.props b/Directory.Packages.props index bdecf324..da338552 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -109,6 +109,7 @@ + diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs index 253de4ba..fa0ab5bb 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs @@ -137,14 +137,23 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) var screenshot = await GetCachedScreenshotAsync(client); var hasScreenshot = screenshot?.Length > 0; - // Get screenshot dimensions for accurate viewport sizing - int width = 0, height = 0; - if (hasScreenshot) + // Get viewport size from agent status (window logical size) + var status = await client.GetStatusAsync(); + var viewportWidth = status?.Device?.WindowWidth ?? 0; + var viewportHeight = status?.Device?.WindowHeight ?? 0; + + // Fallback to screenshot dimensions if status didn't provide window size + if (viewportWidth <= 0 || viewportHeight <= 0) { - (width, height) = GetPngDimensions(screenshot!); + if (hasScreenshot) + { + var (pw, ph) = GetPngDimensions(screenshot!); + viewportWidth = pw; + viewportHeight = ph; + } } - var html = HtmlRenderer.Render(tree, hasScreenshot, width, height); + var html = HtmlRenderer.Render(tree, hasScreenshot, (int)viewportWidth, (int)viewportHeight); return (200, "text/html; charset=utf-8", Encoding.UTF8.GetBytes(html)); } @@ -152,7 +161,6 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) private static (int width, int height) GetPngDimensions(byte[] png) { if (png.Length < 24) return (0, 0); - // PNG IHDR: width at offset 16 (4 bytes big-endian), height at offset 20 int w = (png[16] << 24) | (png[17] << 16) | (png[18] << 8) | png[19]; int h = (png[20] << 24) | (png[21] << 16) | (png[22] << 8) | png[23]; return (w, h); diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css index 2fda1d33..189ed005 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css @@ -8,11 +8,17 @@ body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #1e1e1e; color: #fff; + display: flex; + align-items: center; + justify-content: center; + min-height: 100vh; + overflow: hidden; } #app-viewport { position: relative; overflow: hidden; + transform-origin: center center; } #screenshot { diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js index 39edd08d..0ff2c2fb 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -9,14 +9,38 @@ let gesturePoints = []; let isGesturing = false; let isDragging = false; + let currentScale = 1; + + // ── Zoom to fit ── + function zoomToFit() { + const appW = parseFloat(viewport.dataset.width) || viewport.offsetWidth; + const appH = parseFloat(viewport.dataset.height) || viewport.offsetHeight; + const winW = window.innerWidth; + const winH = window.innerHeight; + + const scaleX = winW / appW; + const scaleY = winH / appH; + currentScale = Math.min(scaleX, scaleY, 1); // never upscale + + viewport.style.transform = `scale(${currentScale})`; + } + + zoomToFit(); + window.addEventListener('resize', zoomToFit); + + // Convert browser coordinates to app logical coordinates (accounting for zoom) + function toAppCoords(clientX, clientY) { + const rect = viewport.getBoundingClientRect(); + const x = (clientX - rect.left) / currentScale; + const y = (clientY - rect.top) / currentScale; + return { x, y }; + } // ── Click → Tap ── viewport.addEventListener('click', async (e) => { if (isDragging) return; - const rect = viewport.getBoundingClientRect(); - const x = e.clientX - rect.left; - const y = e.clientY - rect.top; + const { x, y } = toAppCoords(e.clientX, e.clientY); try { await fetch('/api/tap', { @@ -33,9 +57,7 @@ // ── Wheel → Scroll ── viewport.addEventListener('wheel', async (e) => { e.preventDefault(); - const rect = viewport.getBoundingClientRect(); - const x = e.clientX - rect.left; - const y = e.clientY - rect.top; + const { x, y } = toAppCoords(e.clientX, e.clientY); try { await fetch('/api/scroll', { @@ -51,7 +73,8 @@ // ── Pointer Drag → Gesture ── viewport.addEventListener('pointerdown', (e) => { - gesturePoints = [{ x: e.offsetX, y: e.offsetY, t: Date.now() }]; + const { x, y } = toAppCoords(e.clientX, e.clientY); + gesturePoints = [{ x, y, t: Date.now() }]; isGesturing = true; isDragging = false; viewport.setPointerCapture(e.pointerId); @@ -59,7 +82,8 @@ viewport.addEventListener('pointermove', (e) => { if (!isGesturing) return; - gesturePoints.push({ x: e.offsetX, y: e.offsetY, t: Date.now() }); + const { x, y } = toAppCoords(e.clientX, e.clientY); + gesturePoints.push({ x, y, t: Date.now() }); if (gesturePoints.length > 3) isDragging = true; }); diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html index 37b6c02f..8e17936c 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html @@ -6,7 +6,7 @@ -
+
{{SCREENSHOT}} {{ELEMENTS}}
diff --git a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs new file mode 100644 index 00000000..be68d0e5 --- /dev/null +++ b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs @@ -0,0 +1,257 @@ +using Microsoft.Playwright; +using Xunit; + +namespace Microsoft.Maui.DevFlow.Inspector.Tests; + +/// +/// Playwright integration tests for the DevFlow Web Inspector. +/// Requires a running inspector (maui devflow inspector) and a connected MAUI app. +/// Set INSPECTOR_URL environment variable to override the default http://localhost:5223. +/// +[Collection("Inspector")] +public class InspectorPageTests : IAsyncLifetime +{ + private IPlaywright _playwright = null!; + private IBrowser _browser = null!; + private IBrowserContext _context = null!; + private IPage _page = null!; + + private string BaseUrl => Environment.GetEnvironmentVariable("INSPECTOR_URL") ?? "http://localhost:5223"; + + public async Task InitializeAsync() + { + _playwright = await Playwright.Playwright.CreateAsync(); + _browser = await _playwright.Chromium.LaunchAsync(new() { Headless = true }); + _context = await _browser.NewContextAsync(); + _page = await _context.NewPageAsync(); + } + + public async Task DisposeAsync() + { + await _context.DisposeAsync(); + await _browser.DisposeAsync(); + _playwright.Dispose(); + } + + [Fact] + public async Task ViewportUsesWindowDimensionsFromAgent() + { + await _page.GotoAsync(BaseUrl); + var viewport = _page.Locator("#app-viewport"); + await Expect(viewport).ToBeVisibleAsync(); + + var width = await viewport.GetAttributeAsync("data-width"); + var height = await viewport.GetAttributeAsync("data-height"); + + // Window dimensions should be positive and NOT the old iPhone defaults + var w = double.Parse(width!); + var h = double.Parse(height!); + Assert.True(w > 0, "Viewport width should be positive"); + Assert.True(h > 0, "Viewport height should be positive"); + Assert.NotEqual(390, w); // Not hardcoded iPhone width + Assert.NotEqual(844, h); // Not hardcoded iPhone height + } + + [Fact] + public async Task ViewportScalesToFitBrowserWindow() + { + await _page.GotoAsync(BaseUrl); + var viewport = _page.Locator("#app-viewport"); + + // The viewport should have a CSS transform applied for zoom + var transform = await viewport.EvaluateAsync( + "el => window.getComputedStyle(el).transform"); + + // If the app is larger than the browser, transform should be a matrix (scaled) + // If it fits, transform could be "none" or a scale(1) matrix + Assert.NotNull(transform); + } + + [Fact] + public async Task ScreenshotImageIsPresent() + { + await _page.GotoAsync(BaseUrl); + var screenshot = _page.Locator("#screenshot"); + await Expect(screenshot).ToBeVisibleAsync(); + + var src = await screenshot.GetAttributeAsync("src"); + Assert.Equal("/screenshot.png", src); + } + + [Fact] + public async Task NoInspectorChromeRendered() + { + await _page.GotoAsync(BaseUrl); + + // No toolbar, no connection status — the host inspector tool provides its own chrome + await Expect(_page.Locator("#devflow-toolbar")).ToHaveCountAsync(0); + await Expect(_page.Locator("#btn-back")).ToHaveCountAsync(0); + await Expect(_page.Locator("#connection-status")).ToHaveCountAsync(0); + } + + [Fact] + public async Task ElementsRenderedAsPositionedDivs() + { + await _page.GotoAsync(BaseUrl); + var elements = _page.Locator(".devflow-element"); + var count = await elements.CountAsync(); + Assert.True(count > 0, "Should have at least one element div"); + + // First element should have required data attributes + var first = elements.First; + var id = await first.GetAttributeAsync("data-id"); + var type = await first.GetAttributeAsync("data-type"); + Assert.NotNull(id); + Assert.NotNull(type); + } + + [Fact] + public async Task ElementPositionsMatchAppCoordinates() + { + await _page.GotoAsync(BaseUrl); + + // Find an element with bounds + var positioned = _page.Locator(".devflow-element[style*='left:']"); + var count = await positioned.CountAsync(); + Assert.True(count > 0, "Should have positioned elements"); + + var style = await positioned.First.GetAttributeAsync("style"); + Assert.NotNull(style); + Assert.Contains("position:absolute", style); + Assert.Matches(@"left:\d", style); + Assert.Matches(@"top:\d", style); + } + + [Fact] + public async Task ElementTreeIsNested() + { + await _page.GotoAsync(BaseUrl); + + // Children should be nested inside parent divs + var nested = _page.Locator(".devflow-element > .devflow-element"); + var count = await nested.CountAsync(); + Assert.True(count > 0, "Elements should be nested (parent > child)"); + } + + [Fact] + public async Task DataAttributesUseCamelCase() + { + await _page.GotoAsync(BaseUrl); + + // DevFlow properties use camelCase: isVisible, isEnabled, fullType + var withVisibility = _page.Locator(".devflow-element[data-isVisible]"); + Assert.True(await withVisibility.CountAsync() > 0); + + var withEnabled = _page.Locator(".devflow-element[data-isEnabled]"); + Assert.True(await withEnabled.CountAsync() > 0); + } + + [Fact] + public async Task CssServedSeparately() + { + var response = await _page.APIRequest.GetAsync($"{BaseUrl}/devflow.css"); + Assert.True(response.Ok); + var text = await response.TextAsync(); + Assert.Contains("#app-viewport", text); + Assert.Contains(".devflow-element", text); + // No hover highlighting — the host inspector adds its own + Assert.DoesNotContain(":hover", text); + } + + [Fact] + public async Task ClickSendsTapToAgent() + { + await _page.GotoAsync(BaseUrl); + + // Get the viewport bounding box + var viewport = _page.Locator("#app-viewport"); + var box = await viewport.BoundingBoxAsync(); + Assert.NotNull(box); + + // Take a screenshot before clicking + var screenshotBefore = await _page.Locator("#screenshot").GetAttributeAsync("src"); + + // Click in the middle of the viewport + await viewport.ClickAsync(new() { Position = new() { X = (float)box.Width / 2, Y = (float)box.Height / 2 } }); + + // Wait for screenshot refresh (devflow.js refreshes after tap) + await _page.WaitForTimeoutAsync(500); + + // The screenshot src should have changed (cache-bust query param) + var screenshotAfter = await _page.Locator("#screenshot").GetAttributeAsync("src"); + Assert.NotEqual(screenshotBefore, screenshotAfter); + } + + [Fact] + public async Task ClickOnElementSendsTapAtCorrectCoordinates() + { + await _page.GotoAsync(BaseUrl); + + // Set up request interception to capture tap coordinates + var tapRequests = new List(); + await _page.RouteAsync("**/api/tap", async route => + { + var body = route.Request.PostData; + tapRequests.Add(body ?? ""); + await route.ContinueAsync(); + }); + + // Find an element with positive width and height in style (not -1 or 0) + var allPositioned = _page.Locator(".devflow-element[style*='width:']"); + var count = await allPositioned.CountAsync(); + ILocator? target = null; + + for (int i = 0; i < count; i++) + { + var style = await allPositioned.Nth(i).GetAttributeAsync("style") ?? ""; + // Parse width value — skip elements with -1 or 0 width + var widthMatch = System.Text.RegularExpressions.Regex.Match(style, @"width:(\d+)px"); + var heightMatch = System.Text.RegularExpressions.Regex.Match(style, @"height:(\d+)px"); + if (widthMatch.Success && heightMatch.Success) + { + var w = int.Parse(widthMatch.Groups[1].Value); + var h = int.Parse(heightMatch.Groups[1].Value); + if (w > 10 && h > 10) + { + target = allPositioned.Nth(i); + break; + } + } + } + + if (target == null) + { + // No suitable element found — skip + return; + } + + // Click with force (the div is transparent overlay, not visually rendered) + await target.ClickAsync(new() { Force = true, Timeout = 5000 }); + await _page.WaitForTimeoutAsync(300); + + // Verify a tap request was sent with valid coordinates + Assert.NotEmpty(tapRequests); + var json = System.Text.Json.JsonDocument.Parse(tapRequests[0]); + var x = json.RootElement.GetProperty("x").GetDouble(); + var y = json.RootElement.GetProperty("y").GetDouble(); + Assert.True(x >= 0, $"Tap x should be non-negative, got {x}"); + Assert.True(y >= 0, $"Tap y should be non-negative, got {y}"); + } + + [Fact] + public async Task ScreenshotEndpointReturnsPng() + { + var response = await _page.APIRequest.GetAsync($"{BaseUrl}/screenshot.png"); + Assert.True(response.Ok); + var body = await response.BodyAsync(); + + // PNG magic bytes + Assert.Equal(0x89, body[0]); + Assert.Equal(0x50, body[1]); // P + Assert.Equal(0x4E, body[2]); // N + Assert.Equal(0x47, body[3]); // G + } + + private ILocatorAssertions Expect(ILocator locator) => + Assertions.Expect(locator); +} diff --git a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/Microsoft.Maui.DevFlow.Inspector.Tests.csproj b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/Microsoft.Maui.DevFlow.Inspector.Tests.csproj new file mode 100644 index 00000000..5ef2bbe2 --- /dev/null +++ b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/Microsoft.Maui.DevFlow.Inspector.Tests.csproj @@ -0,0 +1,23 @@ + + + + net10.0 + false + + + + + + + + + + + + + + + + + + diff --git a/tests/Inspector.Playwright/inspector.spec.js b/tests/Inspector.Playwright/inspector.spec.js deleted file mode 100644 index cfae267f..00000000 --- a/tests/Inspector.Playwright/inspector.spec.js +++ /dev/null @@ -1,133 +0,0 @@ -// @ts-check -const { test, expect } = require('@playwright/test'); - -test.describe('DevFlow Inspector HTML output', () => { - - test('page loads with correct viewport dimensions from screenshot', async ({ page }) => { - await page.goto('/'); - - const viewport = page.locator('#app-viewport'); - await expect(viewport).toBeVisible(); - - // Viewport should use actual screenshot dimensions (not hardcoded 390x844) - const style = await viewport.getAttribute('style'); - expect(style).toContain('width:'); - expect(style).toContain('height:'); - - // Should NOT be iPhone defaults - expect(style).not.toContain('width:390px'); - expect(style).not.toContain('height:844px'); - }); - - test('page contains screenshot image', async ({ page }) => { - await page.goto('/'); - - const screenshot = page.locator('#screenshot'); - await expect(screenshot).toBeVisible(); - expect(await screenshot.getAttribute('src')).toBe('/screenshot.png'); - }); - - test('page has no toolbar or inspector chrome', async ({ page }) => { - await page.goto('/'); - - // Should NOT have toolbar elements — the host inspector adds its own - await expect(page.locator('#devflow-toolbar')).toHaveCount(0); - await expect(page.locator('#btn-back')).toHaveCount(0); - await expect(page.locator('#btn-refresh')).toHaveCount(0); - await expect(page.locator('#connection-status')).toHaveCount(0); - }); - - test('elements have no hover highlighting styles', async ({ page }) => { - await page.goto('/'); - - // Check that CSS doesn't include hover outline - const cssResponse = await page.request.get('/devflow.css'); - const cssText = await cssResponse.text(); - expect(cssText).not.toContain(':hover'); - expect(cssText).not.toContain('outline'); - }); - - test('elements rendered as positioned divs with data attributes', async ({ page }) => { - await page.goto('/'); - - const elements = page.locator('.devflow-element'); - const count = await elements.count(); - expect(count).toBeGreaterThan(0); - - // First element should have required data attributes - const first = elements.first(); - expect(await first.getAttribute('data-id')).toBeTruthy(); - expect(await first.getAttribute('data-type')).toBeTruthy(); - }); - - test('elements have correct positioning styles', async ({ page }) => { - await page.goto('/'); - - // Find an element with bounds that has positive dimensions - const elements = page.locator('.devflow-element[style*="left:"]'); - const count = await elements.count(); - expect(count).toBeGreaterThan(0); - - const style = await elements.first().getAttribute('style'); - expect(style).toContain('position:absolute'); - expect(style).toMatch(/left:\d/); - expect(style).toMatch(/top:\d/); - }); - - test('screenshot endpoint returns PNG', async ({ page }) => { - const response = await page.request.get('/screenshot.png'); - expect(response.status()).toBe(200); - expect(response.headers()['content-type']).toBe('image/png'); - - const body = await response.body(); - // PNG magic bytes - expect(body[0]).toBe(0x89); - expect(body[1]).toBe(0x50); // P - expect(body[2]).toBe(0x4E); // N - expect(body[3]).toBe(0x47); // G - }); - - test('CSS served as separate file', async ({ page }) => { - const response = await page.request.get('/devflow.css'); - expect(response.status()).toBe(200); - expect(response.headers()['content-type']).toBe('text/css'); - - const text = await response.text(); - expect(text).toContain('#app-viewport'); - expect(text).toContain('.devflow-element'); - }); - - test('JS served as separate file', async ({ page }) => { - const response = await page.request.get('/devflow.js'); - expect(response.status()).toBe(200); - expect(response.headers()['content-type']).toBe('application/javascript'); - - const text = await response.text(); - expect(text).toContain('app-viewport'); - expect(text).toContain('/api/tap'); - }); - - test('element tree is nested (children inside parents)', async ({ page }) => { - await page.goto('/'); - - // Find a parent element that contains child elements - const nestedParent = page.locator('.devflow-element > .devflow-element'); - const count = await nestedParent.count(); - expect(count).toBeGreaterThan(0); - }); - - test('data attributes use camelCase naming from DevFlow API', async ({ page }) => { - await page.goto('/'); - - // Check attributes directly on elements rather than raw HTML - const elemWithVis = page.locator('.devflow-element[data-isVisible]'); - expect(await elemWithVis.count()).toBeGreaterThan(0); - - const elemWithEnabled = page.locator('.devflow-element[data-isEnabled]'); - expect(await elemWithEnabled.count()).toBeGreaterThan(0); - - // Verify camelCase naming convention - const elemWithFullType = page.locator('.devflow-element[data-fullType]'); - expect(await elemWithFullType.count()).toBeGreaterThan(0); - }); -}); diff --git a/tests/Inspector.Playwright/package-lock.json b/tests/Inspector.Playwright/package-lock.json deleted file mode 100644 index 9b153d18..00000000 --- a/tests/Inspector.Playwright/package-lock.json +++ /dev/null @@ -1,79 +0,0 @@ -{ - "name": "inspector.playwright", - "version": "1.0.0", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "inspector.playwright", - "version": "1.0.0", - "license": "ISC", - "devDependencies": { - "@playwright/test": "^1.60.0" - } - }, - "node_modules/@playwright/test": { - "version": "1.60.0", - "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.60.0.tgz", - "integrity": "sha512-O71yZIbAh/PxDMNGns37GHBIfrVkEVyn+AXyIa5dOTfb4/xNvRWV+Vv/NMbNCtODB/pO7vLlF2OTmMVLhmr7Ag==", - "dev": true, - "license": "Apache-2.0", - "dependencies": { - "playwright": "1.60.0" - }, - "bin": { - "playwright": "cli.js" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/fsevents": { - "version": "2.3.2", - "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", - "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", - "dev": true, - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": "^8.16.0 || ^10.6.0 || >=11.0.0" - } - }, - "node_modules/playwright": { - "version": "1.60.0", - "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.60.0.tgz", - "integrity": "sha512-hheHdokM8cdqCb0lcE3s+zT4t4W+vvjpGxsZlDnikarzx8tSzMebh3UiFtgqwFwnTnjYQcsyMF8ei2mCO/tpeA==", - "dev": true, - "license": "Apache-2.0", - "dependencies": { - "playwright-core": "1.60.0" - }, - "bin": { - "playwright": "cli.js" - }, - "engines": { - "node": ">=18" - }, - "optionalDependencies": { - "fsevents": "2.3.2" - } - }, - "node_modules/playwright-core": { - "version": "1.60.0", - "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.60.0.tgz", - "integrity": "sha512-9bW6zvX/m0lEbgTKJ6YppOKx8H3VOPBMOCFh2irXFOT4BbHgrx5hPjwJYLT40Lu+4qtD36qKc/Hn56StUW57IA==", - "dev": true, - "license": "Apache-2.0", - "bin": { - "playwright-core": "cli.js" - }, - "engines": { - "node": ">=18" - } - } - } -} diff --git a/tests/Inspector.Playwright/package.json b/tests/Inspector.Playwright/package.json deleted file mode 100644 index b1553e12..00000000 --- a/tests/Inspector.Playwright/package.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "name": "inspector.playwright", - "version": "1.0.0", - "description": "", - "main": "index.js", - "scripts": { - "test": "echo \"Error: no test specified\" && exit 1" - }, - "keywords": [], - "author": "", - "license": "ISC", - "type": "commonjs", - "devDependencies": { - "@playwright/test": "^1.60.0" - } -} diff --git a/tests/Inspector.Playwright/playwright.config.js b/tests/Inspector.Playwright/playwright.config.js deleted file mode 100644 index e7709523..00000000 --- a/tests/Inspector.Playwright/playwright.config.js +++ /dev/null @@ -1,10 +0,0 @@ -// @ts-check -const { defineConfig } = require('@playwright/test'); - -module.exports = defineConfig({ - testDir: '.', - timeout: 30000, - use: { - baseURL: 'http://localhost:5223', - }, -}); From dabbc6969489f070836fcee03aa3c2c9ad9a4114 Mon Sep 17 00:00:00 2001 From: Matthew Leibowitz Date: Fri, 22 May 2026 22:53:17 +0200 Subject: [PATCH 04/13] =?UTF-8?q?Integrate=20inspector=20into=20broker=20?= =?UTF-8?q?=E2=80=94=20no=20separate=20command=20needed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The web inspector is now served directly by the broker at http://localhost:19223/inspector/ (or /inspector/{agentId}/ for multi-agent). Since the broker is already running whenever a MAUI app connects, this removes the need for a separate 'maui devflow inspector' command. Changes: - BrokerServer: add /inspector routes that proxy to InspectorServer - InspectorServer: add HandleBrokerRequestAsync + WebSocket relay - Remove standalone 'inspector' CLI subcommand - Make all asset paths relative (works under any URL prefix) - devflow.js: derive basePath from location.pathname - Clean up inspector on agent disconnect Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../DevFlow/Broker/BrokerServer.cs | 121 +++++++++++++++++- .../DevFlow/DevFlowCommands.cs | 41 ------ .../DevFlow/Inspector/HtmlRenderer.cs | 2 +- .../DevFlow/Inspector/InspectorServer.cs | 95 +++++++++++++- .../DevFlow/Inspector/Web/devflow.js | 13 +- .../DevFlow/Inspector/Web/inspector.html | 4 +- .../InspectorPageTests.cs | 7 +- 7 files changed, 226 insertions(+), 57 deletions(-) diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs index 9de62692..93a14857 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs @@ -4,6 +4,7 @@ using System.Net.WebSockets; using System.Text; using System.Text.Json.Nodes; +using Microsoft.Maui.Cli.DevFlow.Inspector; namespace Microsoft.Maui.Cli.DevFlow.Broker; @@ -98,6 +99,13 @@ private async Task HandleRequestAsync(HttpListenerContext context) return; } + // WebSocket upgrade for inspector event relay + if (context.Request.IsWebSocketRequest && path.StartsWith("/inspector", StringComparison.OrdinalIgnoreCase)) + { + await HandleInspectorRoute(context, path); + return; + } + // HTTP endpoints for CLI var (statusCode, body) = (method, path) switch { @@ -108,12 +116,22 @@ private async Task HandleRequestAsync(HttpListenerContext context) }, indented: false)), ("GET", "/api/agents") => (200, HandleListAgents()), ("POST", "/api/shutdown") => HandleShutdown(), - _ => (404, CliJson.SerializeUntyped(new JsonObject - { - ["error"] = "Not found" - }, indented: false)) + _ => (0, "") // handled below for inspector routes }; + // Inspector routes — serve the web inspector for connected agents + if (statusCode == 0) + { + if (path.StartsWith("/inspector", StringComparison.OrdinalIgnoreCase)) + { + await HandleInspectorRoute(context, path); + return; + } + + statusCode = 404; + body = CliJson.SerializeUntyped(new JsonObject { ["error"] = "Not found" }, indented: false); + } + context.Response.StatusCode = statusCode; context.Response.ContentType = "application/json"; context.Response.Headers.Add("Access-Control-Allow-Origin", "*"); @@ -251,6 +269,8 @@ private async Task MonitorAgentConnection(AgentConnection connection) if (_agents.TryRemove(connection.Registration.Id, out _)) { ReleasePort(connection.Registration.Port); + if (_inspectors.TryRemove(connection.Registration.Id, out var inspector)) + inspector.Dispose(); Log($"Agent disconnected: {connection.Registration.AppName}|{connection.Registration.Tfm}"); } } @@ -409,4 +429,97 @@ public void Dispose() _cts?.Dispose(); } private record AgentConnection(AgentRegistration Registration, WebSocket WebSocket); + + // ── Inspector integration ── + + private readonly ConcurrentDictionary _inspectors = new(); + + private async Task HandleInspectorRoute(HttpListenerContext context, string path) + { + // Routes: + // /inspector → list agents with inspector links + // /inspector/{id} → serve inspector HTML for that agent + // /inspector/{id}/... → proxy sub-routes to the per-agent InspectorServer + + var segments = path.TrimStart('/').Split('/', 3); + + if (segments.Length == 1 || (segments.Length == 2 && string.IsNullOrEmpty(segments[1]))) + { + // List agents with inspector links + await ServeAgentListPage(context); + return; + } + + var agentId = segments[1]; + var subPath = segments.Length > 2 ? "/" + segments[2] : "/"; + + // Find the agent + if (!_agents.TryGetValue(agentId, out var connection)) + { + // Try partial match + connection = _agents.Values.FirstOrDefault(a => + a.Registration.Id.StartsWith(agentId, StringComparison.OrdinalIgnoreCase)); + if (connection == null) + { + // If only one agent connected, use it + if (_agents.Count == 1) + connection = _agents.Values.First(); + } + } + + if (connection == null) + { + context.Response.StatusCode = 404; + context.Response.ContentType = "text/plain"; + var msg = Encoding.UTF8.GetBytes($"Agent '{agentId}' not found. Connected agents: {_agents.Count}"); + await context.Response.OutputStream.WriteAsync(msg); + context.Response.Close(); + return; + } + + // Get or create inspector server for this agent + var inspector = _inspectors.GetOrAdd(connection.Registration.Id, _ => + { + var server = new InspectorServer(0, "localhost", connection.Registration.Port); + Log($"Inspector created for agent: {connection.Registration.AppName} (port {connection.Registration.Port})"); + return server; + }); + + // Proxy the request through the inspector's route handler + await inspector.HandleBrokerRequestAsync(context, subPath); + } + + private async Task ServeAgentListPage(HttpListenerContext context) + { + var sb = new System.Text.StringBuilder(); + sb.AppendLine("DevFlow Inspector"); + sb.AppendLine(""); + sb.AppendLine("

DevFlow Inspector

"); + + if (_agents.IsEmpty) + { + sb.AppendLine("

No agents connected. Start a MAUI app with DevFlow enabled.

"); + } + else + { + foreach (var agent in _agents.Values) + { + var reg = agent.Registration; + sb.AppendLine($"
"); + sb.AppendLine($"{reg.AppName}"); + sb.AppendLine($" — {reg.Platform} ({reg.Tfm}) on port {reg.Port}"); + sb.AppendLine($"
"); + } + } + + sb.AppendLine(""); + context.Response.StatusCode = 200; + context.Response.ContentType = "text/html; charset=utf-8"; + var bytes = Encoding.UTF8.GetBytes(sb.ToString()); + context.Response.ContentLength64 = bytes.Length; + await context.Response.OutputStream.WriteAsync(bytes); + context.Response.Close(); + } } diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs index 95c738bb..a42521db 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/DevFlowCommands.cs @@ -1568,47 +1568,6 @@ await MauiScrollAsync(host, port, isJson, mcpCmd.SetAction(async (ctx, ct) => { await Mcp.McpServerHost.RunAsync(); }); devflowCommand.Add(mcpCmd); - // ===== Inspector command ===== - var inspectorPortOption = new Option("--port") { Description = "Inspector server port", DefaultValueFactory = _ => 5223 }; - var inspectorCmd = new Command("inspector", "Start web inspector — serves the app as an interactive HTML page"); - inspectorCmd.Add(inspectorPortOption); - inspectorCmd.SetAction(async (ctx, ct) => - { - var host = ctx.GetValue(agentHostOption)!; - var port = ctx.GetValue(agentPortOption); - var inspectorPort = ctx.GetValue(inspectorPortOption); - - var server = new Inspector.InspectorServer(inspectorPort, host, port); - server.Start(); - - var url = $"http://localhost:{inspectorPort}"; - Console.WriteLine($"DevFlow Inspector running at {url}"); - Console.WriteLine($"Proxying to agent at {host}:{port}"); - Console.WriteLine("Press Ctrl+C to stop."); - - // Try to open browser - try - { - if (OperatingSystem.IsMacOS()) - System.Diagnostics.Process.Start("open", url); - else if (OperatingSystem.IsWindows()) - System.Diagnostics.Process.Start(new System.Diagnostics.ProcessStartInfo(url) { UseShellExecute = true }); - else if (OperatingSystem.IsLinux()) - System.Diagnostics.Process.Start("xdg-open", url); - } - catch { } - - // Wait until cancelled - var tcs = new TaskCompletionSource(); - ct.Register(() => tcs.TrySetResult()); - Console.CancelKeyPress += (_, e) => { e.Cancel = true; tcs.TrySetResult(); }; - await tcs.Task; - - await server.StopAsync(); - server.Dispose(); - }); - devflowCommand.Add(inspectorCmd); - _devflowCommand = devflowCommand; return devflowCommand; diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs index 479d01ff..ee8d8725 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs @@ -43,7 +43,7 @@ public static string Render(List tree, bool hasScreenshot, int scre // Build screenshot tag var screenshotHtml = hasScreenshot - ? "\"App" + ? "\"App" : ""; // Replace template placeholders diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs index fa0ab5bb..12ee0d57 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs @@ -33,6 +33,99 @@ public InspectorServer(int port, string agentHost, int agentPort) _agentPort = agentPort; } + /// + /// Handles an HTTP request from the broker, routing it through the inspector logic. + /// This allows the broker to serve inspector pages without a separate listener. + /// + public async Task HandleBrokerRequestAsync(HttpListenerContext context, string path) + { + try + { + // Handle WebSocket upgrade for /ws/events + if (context.Request.IsWebSocketRequest && path.TrimEnd('/') == "/ws/events") + { + await HandleBrokerWebSocketProxy(context); + return; + } + + var method = context.Request.HttpMethod; + string? body = null; + if (method == "POST" && context.Request.HasEntityBody) + { + using var reader = new System.IO.StreamReader(context.Request.InputStream, context.Request.ContentEncoding); + body = await reader.ReadToEndAsync(); + } + + var request = new HttpRequestInfo { Method = method, Path = path, Body = body }; + var (statusCode, contentType, responseBody) = await RouteAsync(request); + + context.Response.StatusCode = statusCode; + context.Response.ContentType = contentType; + context.Response.Headers.Add("Access-Control-Allow-Origin", "*"); + context.Response.ContentLength64 = responseBody.Length; + await context.Response.OutputStream.WriteAsync(responseBody); + context.Response.Close(); + } + catch (Exception ex) + { + try + { + context.Response.StatusCode = 500; + var msg = Encoding.UTF8.GetBytes($"Inspector error: {ex.Message}"); + await context.Response.OutputStream.WriteAsync(msg); + context.Response.Close(); + } + catch { } + } + } + + /// + /// Proxies a WebSocket connection from the broker to the agent's /ws/events endpoint. + /// + private async Task HandleBrokerWebSocketProxy(HttpListenerContext context) + { + var wsContext = await context.AcceptWebSocketAsync(null); + var clientWs = wsContext.WebSocket; + + using var agentWs = new System.Net.WebSockets.ClientWebSocket(); + var agentUri = new Uri($"ws://{_agentHost}:{_agentPort}/ws/events"); + + try + { + await agentWs.ConnectAsync(agentUri, CancellationToken.None); + } + catch + { + await clientWs.CloseAsync(System.Net.WebSockets.WebSocketCloseStatus.EndpointUnavailable, + "Agent not reachable", CancellationToken.None); + return; + } + + // Relay messages from agent to browser + var buffer = new byte[4096]; + try + { + while (agentWs.State == System.Net.WebSockets.WebSocketState.Open && + clientWs.State == System.Net.WebSockets.WebSocketState.Open) + { + var result = await agentWs.ReceiveAsync(buffer, CancellationToken.None); + if (result.MessageType == System.Net.WebSockets.WebSocketMessageType.Close) break; + + await clientWs.SendAsync( + new ArraySegment(buffer, 0, result.Count), + result.MessageType, result.EndOfMessage, CancellationToken.None); + } + } + catch { } + finally + { + if (clientWs.State == System.Net.WebSockets.WebSocketState.Open) + try { await clientWs.CloseAsync(System.Net.WebSockets.WebSocketCloseStatus.NormalClosure, "", CancellationToken.None); } catch { } + if (agentWs.State == System.Net.WebSockets.WebSocketState.Open) + try { await agentWs.CloseAsync(System.Net.WebSockets.WebSocketCloseStatus.NormalClosure, "", CancellationToken.None); } catch { } + } + } + public void Start() { _cts = new CancellationTokenSource(); @@ -528,7 +621,7 @@ private static async Task WriteResponseAsync(NetworkStream stream, int statusCod await stream.FlushAsync(ct); } - private sealed class HttpRequestInfo + internal sealed class HttpRequestInfo { public string Method { get; init; } = ""; public string Path { get; init; } = ""; diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js index 0ff2c2fb..10f8635a 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -6,6 +6,9 @@ const viewport = document.getElementById('app-viewport'); const screenshot = document.getElementById('screenshot'); + // Determine base path for API calls (handles being served under /inspector/{id}/) + const basePath = location.pathname.replace(/\/$/, ''); + let gesturePoints = []; let isGesturing = false; let isDragging = false; @@ -43,7 +46,7 @@ const { x, y } = toAppCoords(e.clientX, e.clientY); try { - await fetch('/api/tap', { + await fetch(`${basePath}/api/tap`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ x, y }) @@ -60,7 +63,7 @@ const { x, y } = toAppCoords(e.clientX, e.clientY); try { - await fetch('/api/scroll', { + await fetch(`${basePath}/api/scroll`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ x, y, deltaX: e.deltaX, deltaY: e.deltaY }) @@ -98,7 +101,7 @@ if (dist > 20) { try { - await fetch('/api/gesture', { + await fetch(`${basePath}/api/gesture`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ points: gesturePoints }) @@ -118,7 +121,7 @@ async function refreshScreenshot() { await sleep(100); if (screenshot) { - screenshot.src = '/screenshot.png?t=' + Date.now(); + screenshot.src = `${basePath}/screenshot.png?t=` + Date.now(); } } @@ -129,7 +132,7 @@ // ── WebSocket for live updates ── function connectWebSocket() { const protocol = location.protocol === 'https:' ? 'wss:' : 'ws:'; - const ws = new WebSocket(`${protocol}//${location.host}/ws/events`); + const ws = new WebSocket(`${protocol}//${location.host}${basePath}/ws/events`); ws.onmessage = (e) => { try { diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html index 8e17936c..44c84343 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/inspector.html @@ -3,7 +3,7 @@ DevFlow Inspector - +
@@ -11,6 +11,6 @@ {{ELEMENTS}}
- + diff --git a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs index be68d0e5..bd6f97f0 100644 --- a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs +++ b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs @@ -5,8 +5,9 @@ namespace Microsoft.Maui.DevFlow.Inspector.Tests; /// /// Playwright integration tests for the DevFlow Web Inspector. -/// Requires a running inspector (maui devflow inspector) and a connected MAUI app. -/// Set INSPECTOR_URL environment variable to override the default http://localhost:5223. +/// Requires the broker running with a connected MAUI app. +/// The inspector is available at http://localhost:19223/inspector/. +/// Set INSPECTOR_URL environment variable to override the default URL. /// [Collection("Inspector")] public class InspectorPageTests : IAsyncLifetime @@ -16,7 +17,7 @@ public class InspectorPageTests : IAsyncLifetime private IBrowserContext _context = null!; private IPage _page = null!; - private string BaseUrl => Environment.GetEnvironmentVariable("INSPECTOR_URL") ?? "http://localhost:5223"; + private string BaseUrl => Environment.GetEnvironmentVariable("INSPECTOR_URL") ?? "http://localhost:19223/inspector/"; public async Task InitializeAsync() { From 55f117c50ef5fa740acbfa7f12af5d4c44fb6fdd Mon Sep 17 00:00:00 2001 From: Matthew Leibowitz Date: Sat, 23 May 2026 00:24:42 +0200 Subject: [PATCH 05/13] fix(inspector): AJAX refresh, modal screenshot, tap error handling - Replace full page reload with AJAX state polling (/api/state endpoint) to avoid page flash during 3-second refresh interval - Fix FindRootPageId to use last child of Window (topmost/modal page) instead of first child, fixing blank screenshot when modal is showing - Change tap handler to return 200 {ok:false} instead of 404 when no tappable element found at coordinates - Extract RenderElements method for generating element HTML independently - Fix wheel scroll closure bug (capture coords before timeout fires) - Update Playwright tests for flat div structure and AJAX behavior - Add StateEndpoint and AjaxRefresh tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../DevFlow/Inspector/HtmlRenderer.cs | 104 +++++------ .../DevFlow/Inspector/InspectorServer.cs | 154 +++++++++++----- .../DevFlow/Inspector/Web/devflow.css | 14 +- .../DevFlow/Inspector/Web/devflow.js | 167 ++++++++++-------- .../ElementInfo.cs | 3 + .../InspectorPageTests.cs | 79 ++++++--- 6 files changed, 329 insertions(+), 192 deletions(-) diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs index ee8d8725..f83fbbb0 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs @@ -15,31 +15,13 @@ public static class HtmlRenderer { private static string? _templateCache; - public static string Render(List tree, bool hasScreenshot, int screenshotWidth = 0, int screenshotHeight = 0) + public static string Render(List tree, bool hasScreenshot, int screenshotWidth = 0, int screenshotHeight = 0, double density = 1, double elementScale = 1) { var template = GetTemplate(); + var (viewportWidth, viewportHeight) = ComputeViewportSize(tree, screenshotWidth, screenshotHeight); - // Use screenshot dimensions as viewport size (most reliable), - // fall back to root element bounds, then default - double viewportWidth, viewportHeight; - if (screenshotWidth > 0 && screenshotHeight > 0) - { - viewportWidth = screenshotWidth; - viewportHeight = screenshotHeight; - } - else - { - var rootBounds = tree.Count > 0 ? tree[0].Bounds : null; - viewportWidth = rootBounds is { Width: > 0 } ? rootBounds.Width : 800; - viewportHeight = rootBounds is { Height: > 0 } ? rootBounds.Height : 600; - } - - // Build the elements HTML - var elements = new StringBuilder(); - foreach (var element in tree) - { - RenderElement(elements, element, 4); - } + // Build the elements HTML (flat list — all elements use window-absolute bounds) + var elementsHtml = RenderElements(tree, elementScale); // Build screenshot tag var screenshotHtml = hasScreenshot @@ -50,12 +32,39 @@ public static string Render(List tree, bool hasScreenshot, int scre var html = template .Replace("{{VIEWPORT_WIDTH}}", viewportWidth.ToString("F0")) .Replace("{{VIEWPORT_HEIGHT}}", viewportHeight.ToString("F0")) + .Replace("{{DENSITY}}", density.ToString("F1")) + .Replace("{{ELEMENT_SCALE}}", elementScale.ToString("F4")) .Replace("{{SCREENSHOT}}", screenshotHtml) - .Replace("{{ELEMENTS}}", elements.ToString()); + .Replace("{{ELEMENTS}}", elementsHtml); return html; } + /// + /// Renders just the element divs (no template wrapping) for AJAX state updates. + /// + public static string RenderElements(List tree, double elementScale = 1) + { + var sb = new StringBuilder(); + foreach (var element in tree) + { + RenderElementsFlat(sb, element, elementScale); + } + return sb.ToString(); + } + + private static (double width, double height) ComputeViewportSize(List tree, int screenshotWidth, int screenshotHeight) + { + if (screenshotWidth > 0 && screenshotHeight > 0) + return (screenshotWidth, screenshotHeight); + + var rootBounds = tree.Count > 0 ? tree[0].Bounds : null; + return ( + rootBounds is { Width: > 0 } ? rootBounds.Width : 800, + rootBounds is { Height: > 0 } ? rootBounds.Height : 600 + ); + } + private static string GetTemplate() { if (_templateCache != null) return _templateCache; @@ -69,19 +78,30 @@ private static string GetTemplate() return _templateCache; } - private static void RenderElement(StringBuilder sb, ElementInfo element, int indent) + /// + /// Renders all elements as flat siblings (no nesting) using window-absolute bounds. + /// + private static void RenderElementsFlat(StringBuilder sb, ElementInfo element, double scale) { - var pad = new string(' ', indent); - - // Build style for positioning - var style = new StringBuilder("position:absolute;"); - if (element.Bounds != null) + RenderSingleElement(sb, element, scale); + if (element.Children != null) { - style.Append($"left:{element.Bounds.X:F0}px;"); - style.Append($"top:{element.Bounds.Y:F0}px;"); - style.Append($"width:{element.Bounds.Width:F0}px;"); - style.Append($"height:{element.Bounds.Height:F0}px;"); + foreach (var child in element.Children) + { + RenderElementsFlat(sb, child, scale); + } } + } + + private static void RenderSingleElement(StringBuilder sb, ElementInfo element, double scale) + { + // Build style for positioning using window-absolute bounds + // (windowBounds is absolute within the window; bounds is relative to parent) + var bounds = element.WindowBounds ?? element.Bounds; + if (bounds == null || (bounds.Width <= 0 && bounds.Height <= 0)) + return; // Skip elements with no meaningful bounds + + var style = $"position:absolute;left:{bounds.X * scale:F0}px;top:{bounds.Y * scale:F0}px;width:{bounds.Width * scale:F0}px;height:{bounds.Height * scale:F0}px;"; // Build data attributes var attrs = new StringBuilder(); @@ -115,23 +135,7 @@ private static void RenderElement(StringBuilder sb, ElementInfo element, int ind if (!string.IsNullOrEmpty(element.NativeType)) attrs.Append($" data-nativeType=\"{Escape(element.NativeType)}\""); - var hasChildren = element.Children is { Count: > 0 }; - - sb.Append($"{pad}
"); - - if (hasChildren) - { - sb.AppendLine(); - foreach (var child in element.Children!) - { - RenderElement(sb, child, indent + 2); - } - sb.AppendLine($"{pad}
"); - } - else - { - sb.AppendLine("
"); - } + sb.AppendLine($"
"); } private static string Escape(string value) => HttpUtility.HtmlAttributeEncode(value); diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs index 12ee0d57..fd860666 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs @@ -22,6 +22,7 @@ public sealed class InspectorServer : IDisposable private readonly int _agentPort; private byte[]? _cachedScreenshot; private DateTime _screenshotCacheTime; + private string? _rootPageId; private static readonly TimeSpan ScreenshotCacheDuration = TimeSpan.FromMilliseconds(200); public int Port => _port; @@ -199,6 +200,7 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) "GET" => request.Path switch { "/" or "" => await HandleRootAsync(), + "/api/state" => await HandleStateAsync(), "/screenshot.png" => await HandleScreenshotAsync(), "/devflow.js" => HandleEmbeddedFile("devflow.js", "application/javascript"), "/devflow.css" => HandleEmbeddedFile("devflow.css", "text/css"), @@ -227,29 +229,79 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) { using var client = new AgentClient(_agentHost, _agentPort); var tree = await client.GetTreeAsync(); - var screenshot = await GetCachedScreenshotAsync(client); - var hasScreenshot = screenshot?.Length > 0; - // Get viewport size from agent status (window logical size) - var status = await client.GetStatusAsync(); - var viewportWidth = status?.Device?.WindowWidth ?? 0; - var viewportHeight = status?.Device?.WindowHeight ?? 0; + // Find the root page element (first child of Window with content). + // On Mac Catalyst, the default screenshot captures the full screen but element + // bounds are relative to the page content. By screenshotting the page element + // directly we get a 1:1 match between pixel coordinates and element bounds. + var rootPageId = FindRootPageId(tree); + _rootPageId = rootPageId; + var screenshot = await GetCachedScreenshotAsync(client, rootPageId); + var hasScreenshot = screenshot?.Length > 0; - // Fallback to screenshot dimensions if status didn't provide window size - if (viewportWidth <= 0 || viewportHeight <= 0) + double viewportWidth = 800, viewportHeight = 600; + if (hasScreenshot) { - if (hasScreenshot) - { - var (pw, ph) = GetPngDimensions(screenshot!); - viewportWidth = pw; - viewportHeight = ph; - } + var (pw, ph) = GetPngDimensions(screenshot!); + viewportWidth = pw; + viewportHeight = ph; } - var html = HtmlRenderer.Render(tree, hasScreenshot, (int)viewportWidth, (int)viewportHeight); + var html = HtmlRenderer.Render(tree, hasScreenshot, (int)viewportWidth, (int)viewportHeight, 1, 1); return (200, "text/html; charset=utf-8", Encoding.UTF8.GetBytes(html)); } + /// + /// Returns JSON state for AJAX polling: screenshot (as timestamped URL) + element divs HTML. + /// This avoids full page reload flash. + /// + private async Task<(int, string, byte[])> HandleStateAsync() + { + using var client = new AgentClient(_agentHost, _agentPort); + var tree = await client.GetTreeAsync(); + + var rootPageId = FindRootPageId(tree); + _rootPageId = rootPageId; + _cachedScreenshot = null; // force fresh screenshot + var screenshot = await GetCachedScreenshotAsync(client, rootPageId); + var hasScreenshot = screenshot?.Length > 0; + + double viewportWidth = 800, viewportHeight = 600; + if (hasScreenshot) + { + var (pw, ph) = GetPngDimensions(screenshot!); + viewportWidth = pw; + viewportHeight = ph; + } + + var elementsHtml = HtmlRenderer.RenderElements(tree, 1); + var timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + + var json = JsonSerializer.Serialize(new + { + screenshotUrl = $"screenshot.png?t={timestamp}", + elements = elementsHtml, + viewportWidth, + viewportHeight + }); + + return (200, "application/json", Encoding.UTF8.GetBytes(json)); + } + + /// + /// Finds the ID of the topmost page element in the tree. + /// When a modal page is showing, it appears as a later child of the Window, + /// so we take the last child which is the topmost visible page. + /// + private static string? FindRootPageId(List tree) + { + if (tree.Count == 0) return null; + var window = tree[0]; + if (window.Children is not { Count: > 0 }) return null; + // Last child is the topmost (modal pages are added after the shell) + return window.Children[^1].Id; + } + /// Reads width/height from PNG IHDR chunk (bytes 16-23). private static (int width, int height) GetPngDimensions(byte[] png) { @@ -262,7 +314,7 @@ private static (int width, int height) GetPngDimensions(byte[] png) private async Task<(int, string, byte[])> HandleScreenshotAsync() { using var client = new AgentClient(_agentHost, _agentPort); - var png = await GetCachedScreenshotAsync(client); + var png = await GetCachedScreenshotAsync(client, _rootPageId); if (png == null || png.Length == 0) return (404, "text/plain", Encoding.UTF8.GetBytes("No screenshot available")); return (200, "image/png", png); @@ -281,12 +333,12 @@ private static (int width, int height) GetPngDimensions(byte[] png) return (200, contentType, ms.ToArray()); } - private async Task GetCachedScreenshotAsync(AgentClient client) + private async Task GetCachedScreenshotAsync(AgentClient client, string? elementId = null) { if (_cachedScreenshot != null && DateTime.UtcNow - _screenshotCacheTime < ScreenshotCacheDuration) return _cachedScreenshot; - _cachedScreenshot = await client.ScreenshotAsync(); + _cachedScreenshot = await client.ScreenshotAsync(elementId: elementId); _screenshotCacheTime = DateTime.UtcNow; return _cachedScreenshot; } @@ -310,21 +362,27 @@ private static (int width, int height) GetPngDimensions(byte[] png) using var client = new AgentClient(_agentHost, _agentPort); var hitResult = await client.HitTestAsync(x, y); - // Parse hit-test result to get element ID + // Parse hit-test result — response is { elements: [{ id, ... }, ...] } using var hitDoc = JsonDocument.Parse(hitResult); - if (hitDoc.RootElement.TryGetProperty("id", out var idProp)) + var elements = hitDoc.RootElement.GetProperty("elements"); + if (elements.GetArrayLength() > 0) { - var elementId = idProp.GetString(); - if (!string.IsNullOrEmpty(elementId)) + // Try elements from most specific to most general until one accepts tap + for (int i = 0; i < elements.GetArrayLength(); i++) { - var success = await client.TapAsync(elementId); - _cachedScreenshot = null; // invalidate cache - return success - ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) - : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + var elementId = elements[i].GetProperty("id").GetString(); + if (!string.IsNullOrEmpty(elementId)) + { + var success = await client.TapAsync(elementId); + if (success) + { + _cachedScreenshot = null; + return (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")); + } + } } } - return (404, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"No element at coordinates\"}")); + return (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false,\"reason\":\"No tappable element at coordinates\"}")); } // Support elementId-based tap @@ -355,24 +413,40 @@ private static (int width, int height) GetPngDimensions(byte[] png) var deltaX = root.TryGetProperty("deltaX", out var dxProp) ? dxProp.GetDouble() : 0; var deltaY = root.TryGetProperty("deltaY", out var dyProp) ? dyProp.GetDouble() : 0; - string? elementId = null; - // If coordinates provided, hit-test to find the scrollable element + // If coordinates provided, hit-test and try each element for scroll if (root.TryGetProperty("x", out var xProp) && root.TryGetProperty("y", out var yProp)) { - using var hitClient = new AgentClient(_agentHost, _agentPort); - var hitResult = await hitClient.HitTestAsync(xProp.GetDouble(), yProp.GetDouble()); + using var client = new AgentClient(_agentHost, _agentPort); + var hitResult = await client.HitTestAsync(xProp.GetDouble(), yProp.GetDouble()); using var hitDoc = JsonDocument.Parse(hitResult); - if (hitDoc.RootElement.TryGetProperty("id", out var idProp)) - elementId = idProp.GetString(); + var elements = hitDoc.RootElement.GetProperty("elements"); + + // Try each element from most specific to general until one accepts scroll + for (int i = 0; i < elements.GetArrayLength(); i++) + { + var elementId = elements[i].GetProperty("id").GetString(); + if (!string.IsNullOrEmpty(elementId)) + { + var success = await client.ScrollAsync(elementId: elementId, deltaX: deltaX, deltaY: deltaY); + if (success) + { + _cachedScreenshot = null; + return (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")); + } + } + } } - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.ScrollAsync(elementId: elementId, deltaX: deltaX, deltaY: deltaY); - _cachedScreenshot = null; - return success - ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) - : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + // Fallback: scroll without element target + { + using var client = new AgentClient(_agentHost, _agentPort); + var success = await client.ScrollAsync(deltaX: deltaX, deltaY: deltaY); + _cachedScreenshot = null; + return success + ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) + : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); + } } private async Task<(int, string, byte[])> HandleProxyGestureAsync(string? body) diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css index 189ed005..d7911a47 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.css @@ -8,25 +8,15 @@ body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: #1e1e1e; color: #fff; - display: flex; - align-items: center; - justify-content: center; - min-height: 100vh; - overflow: hidden; + overflow: auto; } #app-viewport { position: relative; - overflow: hidden; - transform-origin: center center; } #screenshot { - position: absolute; - top: 0; - left: 0; - width: 100%; - height: 100%; + display: block; pointer-events: none; user-select: none; } diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js index 10f8635a..2a945d78 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -12,31 +12,62 @@ let gesturePoints = []; let isGesturing = false; let isDragging = false; - let currentScale = 1; + let refreshInProgress = false; - // ── Zoom to fit ── - function zoomToFit() { - const appW = parseFloat(viewport.dataset.width) || viewport.offsetWidth; - const appH = parseFloat(viewport.dataset.height) || viewport.offsetHeight; - const winW = window.innerWidth; - const winH = window.innerHeight; + // Convert browser coordinates to app logical coordinates + function toAppCoords(clientX, clientY) { + const rect = viewport.getBoundingClientRect(); + return { x: clientX - rect.left, y: clientY - rect.top }; + } - const scaleX = winW / appW; - const scaleY = winH / appH; - currentScale = Math.min(scaleX, scaleY, 1); // never upscale + // Refresh state via AJAX (no full page reload — avoids flash) + async function refreshState() { + if (refreshInProgress) return; + refreshInProgress = true; + try { + const resp = await fetch(`${basePath}/api/state`); + if (!resp.ok) return; + const state = await resp.json(); - viewport.style.transform = `scale(${currentScale})`; - } + // Update screenshot without flash + if (screenshot && state.screenshotUrl) { + screenshot.src = state.screenshotUrl; + } - zoomToFit(); - window.addEventListener('resize', zoomToFit); + // Update viewport size if changed + if (state.viewportWidth && state.viewportHeight) { + viewport.style.width = state.viewportWidth + 'px'; + viewport.style.height = state.viewportHeight + 'px'; + viewport.dataset.width = state.viewportWidth; + viewport.dataset.height = state.viewportHeight; + } - // Convert browser coordinates to app logical coordinates (accounting for zoom) - function toAppCoords(clientX, clientY) { - const rect = viewport.getBoundingClientRect(); - const x = (clientX - rect.left) / currentScale; - const y = (clientY - rect.top) / currentScale; - return { x, y }; + // Replace element divs (remove old, insert new) + const oldElements = viewport.querySelectorAll('.devflow-element'); + oldElements.forEach(el => el.remove()); + + if (state.elements) { + const temp = document.createElement('div'); + temp.innerHTML = state.elements; + while (temp.firstChild) { + viewport.appendChild(temp.firstChild); + } + } + } catch (err) { + console.error('State refresh failed:', err); + } finally { + refreshInProgress = false; + } + } + + // Debounced refresh — coalesce rapid calls + let refreshTimer = null; + function scheduleRefresh(delayMs) { + if (refreshTimer) clearTimeout(refreshTimer); + refreshTimer = setTimeout(() => { + refreshTimer = null; + refreshState(); + }, delayMs || 300); } // ── Click → Tap ── @@ -51,27 +82,43 @@ headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ x, y }) }); - await refreshScreenshot(); + scheduleRefresh(400); } catch (err) { console.error('Tap failed:', err); } }); // ── Wheel → Scroll ── - viewport.addEventListener('wheel', async (e) => { + let scrollAccumX = 0, scrollAccumY = 0; + let scrollFlushTimer = null; + let lastScrollX = 0, lastScrollY = 0; + + viewport.addEventListener('wheel', (e) => { e.preventDefault(); - const { x, y } = toAppCoords(e.clientX, e.clientY); + scrollAccumX += e.deltaX; + scrollAccumY += e.deltaY; + lastScrollX = e.clientX; + lastScrollY = e.clientY; + + if (scrollFlushTimer) clearTimeout(scrollFlushTimer); + scrollFlushTimer = setTimeout(async () => { + const { x, y } = toAppCoords(lastScrollX, lastScrollY); + const dx = scrollAccumX, dy = scrollAccumY; + scrollAccumX = 0; + scrollAccumY = 0; + scrollFlushTimer = null; - try { - await fetch(`${basePath}/api/scroll`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ x, y, deltaX: e.deltaX, deltaY: e.deltaY }) - }); - await refreshScreenshot(); - } catch (err) { - console.error('Scroll failed:', err); - } + try { + await fetch(`${basePath}/api/scroll`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ x, y, deltaX: dx, deltaY: dy }) + }); + scheduleRefresh(300); + } catch (err) { + console.error('Scroll failed:', err); + } + }, 100); }, { passive: false }); // ── Pointer Drag → Gesture ── @@ -106,7 +153,7 @@ headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ points: gesturePoints }) }); - await refreshScreenshot(); + scheduleRefresh(300); } catch (err) { console.error('Gesture failed:', err); } @@ -117,41 +164,21 @@ setTimeout(() => { isDragging = false; }, 50); }); - // ── Screenshot refresh ── - async function refreshScreenshot() { - await sleep(100); - if (screenshot) { - screenshot.src = `${basePath}/screenshot.png?t=` + Date.now(); + // ── Periodic refresh for app-side changes (AJAX, no flash) ── + let pollInterval = setInterval(() => { + if (!document.hidden && !refreshTimer) { + refreshState(); } - } - - function sleep(ms) { - return new Promise(resolve => setTimeout(resolve, ms)); - } - - // ── WebSocket for live updates ── - function connectWebSocket() { - const protocol = location.protocol === 'https:' ? 'wss:' : 'ws:'; - const ws = new WebSocket(`${protocol}//${location.host}${basePath}/ws/events`); - - ws.onmessage = (e) => { - try { - const event = JSON.parse(e.data); - if (event.type === 'treeChange' || event.type === 'navigation') { - clearTimeout(ws._refreshTimer); - ws._refreshTimer = setTimeout(() => location.reload(), 200); - } - } catch { } - }; - - ws.onclose = () => { - setTimeout(connectWebSocket, 2000); - }; - - ws.onerror = () => { - ws.close(); - }; - } - - connectWebSocket(); + }, 3000); + + document.addEventListener('visibilitychange', () => { + if (document.hidden) { + clearInterval(pollInterval); + pollInterval = null; + } else if (!pollInterval) { + pollInterval = setInterval(() => { + if (!refreshTimer) refreshState(); + }, 3000); + } + }); })(); diff --git a/src/DevFlow/Microsoft.Maui.DevFlow.Driver/ElementInfo.cs b/src/DevFlow/Microsoft.Maui.DevFlow.Driver/ElementInfo.cs index 17c5ad66..a954a593 100644 --- a/src/DevFlow/Microsoft.Maui.DevFlow.Driver/ElementInfo.cs +++ b/src/DevFlow/Microsoft.Maui.DevFlow.Driver/ElementInfo.cs @@ -87,6 +87,9 @@ public ElementStateInfo State [JsonPropertyName("bounds")] public BoundsInfo? Bounds { get; set; } + [JsonPropertyName("windowBounds")] + public BoundsInfo? WindowBounds { get; set; } + [JsonPropertyName("gestures")] public List? Gestures { get; set; } diff --git a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs index bd6f97f0..9a7fe737 100644 --- a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs +++ b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs @@ -54,18 +54,16 @@ public async Task ViewportUsesWindowDimensionsFromAgent() } [Fact] - public async Task ViewportScalesToFitBrowserWindow() + public async Task ViewportHasFixedDimensions() { await _page.GotoAsync(BaseUrl); var viewport = _page.Locator("#app-viewport"); - // The viewport should have a CSS transform applied for zoom - var transform = await viewport.EvaluateAsync( - "el => window.getComputedStyle(el).transform"); - - // If the app is larger than the browser, transform should be a matrix (scaled) - // If it fits, transform could be "none" or a scale(1) matrix - Assert.NotNull(transform); + // Viewport should have explicit width/height style + var style = await viewport.GetAttributeAsync("style"); + Assert.NotNull(style); + Assert.Contains("width:", style); + Assert.Contains("height:", style); } [Fact] @@ -76,7 +74,7 @@ public async Task ScreenshotImageIsPresent() await Expect(screenshot).ToBeVisibleAsync(); var src = await screenshot.GetAttributeAsync("src"); - Assert.Equal("/screenshot.png", src); + Assert.Contains("screenshot.png", src); } [Fact] @@ -124,14 +122,16 @@ public async Task ElementPositionsMatchAppCoordinates() } [Fact] - public async Task ElementTreeIsNested() + public async Task ElementTreeIsFlatNotNested() { await _page.GotoAsync(BaseUrl); - // Children should be nested inside parent divs + // All elements should be direct children of viewport (flat rendering) + var directChildren = _page.Locator("#app-viewport > .devflow-element"); var nested = _page.Locator(".devflow-element > .devflow-element"); - var count = await nested.CountAsync(); - Assert.True(count > 0, "Elements should be nested (parent > child)"); + + Assert.True(await directChildren.CountAsync() > 0, "Should have flat element divs"); + Assert.Equal(0, await nested.CountAsync()); } [Fact] @@ -169,18 +169,15 @@ public async Task ClickSendsTapToAgent() var box = await viewport.BoundingBoxAsync(); Assert.NotNull(box); - // Take a screenshot before clicking - var screenshotBefore = await _page.Locator("#screenshot").GetAttributeAsync("src"); - // Click in the middle of the viewport await viewport.ClickAsync(new() { Position = new() { X = (float)box.Width / 2, Y = (float)box.Height / 2 } }); - // Wait for screenshot refresh (devflow.js refreshes after tap) - await _page.WaitForTimeoutAsync(500); + // Wait for AJAX refresh (devflow.js refreshes after tap via /api/state) + await _page.WaitForTimeoutAsync(1000); - // The screenshot src should have changed (cache-bust query param) + // The screenshot src should have a cache-busting timestamp var screenshotAfter = await _page.Locator("#screenshot").GetAttributeAsync("src"); - Assert.NotEqual(screenshotBefore, screenshotAfter); + Assert.Contains("?t=", screenshotAfter); } [Fact] @@ -253,6 +250,48 @@ public async Task ScreenshotEndpointReturnsPng() Assert.Equal(0x47, body[3]); // G } + [Fact] + public async Task StateEndpointReturnsJsonWithElements() + { + var response = await _page.APIRequest.GetAsync($"{BaseUrl}api/state"); + Assert.True(response.Ok); + var text = await response.TextAsync(); + var json = System.Text.Json.JsonDocument.Parse(text); + + Assert.True(json.RootElement.TryGetProperty("screenshotUrl", out var url)); + Assert.Contains("screenshot.png", url.GetString()); + + Assert.True(json.RootElement.TryGetProperty("elements", out var elements)); + Assert.Contains("devflow-element", elements.GetString()); + + Assert.True(json.RootElement.TryGetProperty("viewportWidth", out var vw)); + Assert.True(vw.GetDouble() > 0); + + Assert.True(json.RootElement.TryGetProperty("viewportHeight", out var vh)); + Assert.True(vh.GetDouble() > 0); + } + + [Fact] + public async Task AjaxRefreshUpdatesElementsWithoutReload() + { + await _page.GotoAsync(BaseUrl); + + // Count initial elements + var initialCount = await _page.Locator(".devflow-element").CountAsync(); + Assert.True(initialCount > 0); + + // Trigger a refresh via JS (simulating what the polling does) + await _page.EvaluateAsync(@"async () => { + const basePath = location.pathname.replace(/\/$/, ''); + const resp = await fetch(basePath + '/api/state'); + return resp.ok; + }"); + + // Elements should still exist (page was not reloaded) + var afterCount = await _page.Locator(".devflow-element").CountAsync(); + Assert.True(afterCount > 0); + } + private ILocatorAssertions Expect(ILocator locator) => Assertions.Expect(locator); } From 4fbd442a2785c4ead1436ccc4b06e574352e08d0 Mon Sep 17 00:00:00 2001 From: Matthew Leibowitz Date: Sat, 23 May 2026 00:26:47 +0200 Subject: [PATCH 06/13] fix(inspector): keyed DOM diff preserves hover/selection state Replace naive remove-all/insert-all element refresh with a keyed diff based on data-id. Elements that haven't changed stay in the DOM, preserving browser hover state, inspector selection, and DevTools highlighting. Only style/attributes are updated in-place when they differ. New elements are inserted, removed ones are deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../DevFlow/Inspector/Web/devflow.js | 91 +++++++++++++++++-- 1 file changed, 82 insertions(+), 9 deletions(-) diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js index 2a945d78..728a1d21 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -42,16 +42,9 @@ viewport.dataset.height = state.viewportHeight; } - // Replace element divs (remove old, insert new) - const oldElements = viewport.querySelectorAll('.devflow-element'); - oldElements.forEach(el => el.remove()); - + // Smart DOM diff — only update elements that changed, preserving hover/selection if (state.elements) { - const temp = document.createElement('div'); - temp.innerHTML = state.elements; - while (temp.firstChild) { - viewport.appendChild(temp.firstChild); - } + patchElements(state.elements); } } catch (err) { console.error('State refresh failed:', err); @@ -60,6 +53,86 @@ } } + // Keyed DOM diff: match elements by data-id, update in-place if changed + function patchElements(newHtml) { + // Parse new elements into a temp container + const temp = document.createElement('div'); + temp.innerHTML = newHtml; + + // Build map of new elements by data-id + const newEls = temp.querySelectorAll('.devflow-element'); + const newMap = new Map(); + const newOrder = []; + newEls.forEach(el => { + const id = el.getAttribute('data-id'); + if (id) { + newMap.set(id, el); + newOrder.push(id); + } + }); + + // Build map of existing elements + const oldEls = viewport.querySelectorAll('.devflow-element'); + const oldMap = new Map(); + oldEls.forEach(el => { + const id = el.getAttribute('data-id'); + if (id) oldMap.set(id, el); + }); + + // Remove elements that no longer exist + oldMap.forEach((el, id) => { + if (!newMap.has(id)) { + el.remove(); + } + }); + + // Update existing elements in-place or insert new ones + let prevEl = screenshot; // insert after screenshot + for (const id of newOrder) { + const newEl = newMap.get(id); + const oldEl = oldMap.get(id); + + if (oldEl) { + // Update only if style or attributes changed + if (oldEl.getAttribute('style') !== newEl.getAttribute('style')) { + oldEl.setAttribute('style', newEl.getAttribute('style')); + } + // Sync data attributes + syncDataAttrs(oldEl, newEl); + // Ensure correct order + if (prevEl && prevEl.nextSibling !== oldEl) { + prevEl.after(oldEl); + } + prevEl = oldEl; + } else { + // New element — insert after previous + const clone = newEl.cloneNode(true); + if (prevEl) { + prevEl.after(clone); + } else { + viewport.appendChild(clone); + } + prevEl = clone; + } + } + } + + // Sync data-* attributes from src to dst without replacing the element + function syncDataAttrs(dst, src) { + // Remove old data attrs not in src + for (const attr of [...dst.attributes]) { + if (attr.name.startsWith('data-') && !src.hasAttribute(attr.name)) { + dst.removeAttribute(attr.name); + } + } + // Set/update from src + for (const attr of src.attributes) { + if (attr.name.startsWith('data-') && dst.getAttribute(attr.name) !== attr.value) { + dst.setAttribute(attr.name, attr.value); + } + } + } + // Debounced refresh — coalesce rapid calls let refreshTimer = null; function scheduleRefresh(delayMs) { From aebe914f614181f1ccfb4f9e2f5939bbb97a8acf Mon Sep 17 00:00:00 2001 From: Rui Marinho Date: Sat, 30 May 2026 01:39:43 +0100 Subject: [PATCH 07/13] fix(inspector): address PR review feedback (security, correctness, tests) Security hardening: - BrokerServer: replace wildcard CORS with localhost-only Origin mirroring; validate Origin BEFORE dispatching POST handlers (the previous check ran after HandleShutdown(), so a cross-origin POST still tore down the broker even though we returned 403). Verified end-to-end with curl. - BrokerServer: reject non-loopback Origin on /ws/agent WebSocket upgrade. - InspectorServer: drop "Access-Control-Allow-Origin: *"; same-origin only. - InspectorServer: enforce Origin on POST endpoints and /ws/v1/ui/events WebSocket proxy (CSRF / cross-origin event subscription). - Add shared LocalOriginValidator helper with unit tests. Correctness: - InspectorServer: cap request bodies at 1 MiB, return 413 on oversize (both broker-hosted and standalone TCP paths), with per-read 10 s timeout on body reads to prevent slow-drip resource exhaustion. - InspectorServer: share a single AgentClient (and dispose it) instead of constructing one per request; gate the screenshot cache with a lock. - InspectorServer: fix WS proxy URL (/ws/events -> /ws/v1/ui/events) and wire lifetime cancellation through _lifetimeCts so the proxy actually tears down on dispose. - InspectorServer: read request bodies as raw bytes once and decode as UTF-8 (the prior ASCII path mangled non-ASCII fill text). - InspectorServer: validate PNG signature + reject negative dims before trusting embedded dimensions. - HtmlRenderer: format all numeric attributes with InvariantCulture and cache the template via Lazy (ExecutionAndPublication). - BrokerServer: HtmlEncode AppName/Platform/Tfm and UrlEncode agent ids in the agent list page (stored XSS via agent metadata). - BrokerServer: dispose hosted inspectors on shutdown. Tests: - New InspectorHtmlRendererTests cover XSS escaping and invariant-culture number formatting (forced via pt-PT comma decimal). - New LocalOriginValidatorTests cover loopback / hostile / malformed Origin. - InspectorPageTests now compose URLs via Uri and parse doubles with InvariantCulture; default URL points at the broker single-agent fallback. Build: - Directory.Build.props: scope false to Apple TFMs (ios/maccatalyst/macos/tvos) instead of repo-wide. 558 unit tests pass; broker security boundaries verified live via curl. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Directory.Build.props | 10 +- .../InspectorHtmlRendererTests.cs | 92 ++++++ .../LocalOriginValidatorTests.cs | 36 +++ .../DevFlow/Broker/BrokerServer.cs | 45 ++- .../DevFlow/Inspector/HtmlRenderer.cs | 27 +- .../DevFlow/Inspector/InspectorServer.cs | 291 +++++++++++++----- .../DevFlow/Inspector/LocalOriginValidator.cs | 33 ++ .../InspectorPageTests.cs | 20 +- 8 files changed, 453 insertions(+), 101 deletions(-) create mode 100644 src/Cli/Microsoft.Maui.Cli.UnitTests/InspectorHtmlRendererTests.cs create mode 100644 src/Cli/Microsoft.Maui.Cli.UnitTests/LocalOriginValidatorTests.cs create mode 100644 src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/LocalOriginValidator.cs diff --git a/Directory.Build.props b/Directory.Build.props index b44030d5..92dbebc1 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -22,7 +22,15 @@ false true - + + + + false diff --git a/src/Cli/Microsoft.Maui.Cli.UnitTests/InspectorHtmlRendererTests.cs b/src/Cli/Microsoft.Maui.Cli.UnitTests/InspectorHtmlRendererTests.cs new file mode 100644 index 00000000..e8d90ebc --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli.UnitTests/InspectorHtmlRendererTests.cs @@ -0,0 +1,92 @@ +using System.Globalization; +using Microsoft.Maui.Cli.DevFlow.Inspector; +using Microsoft.Maui.DevFlow.Driver; +using Xunit; + +namespace Microsoft.Maui.Cli.UnitTests; + +public class InspectorHtmlRendererTests +{ + [Fact] + public void RenderElements_EscapesHtmlInTextAttribute() + { + var tree = new List + { + new() + { + Id = "e1", + Type = "Label", + Text = "", + Bounds = new BoundsInfo { X = 0, Y = 0, Width = 10, Height = 10 }, + }, + }; + + var html = HtmlRenderer.RenderElements(tree); + + // HtmlAttributeEncode encodes <, &, " (but not >). The key invariant is + // that raw " + { + new() + { + Id = "e1", + Type = "Label", + Opacity = 0.5, + Bounds = new BoundsInfo { X = 0, Y = 0, Width = 10, Height = 10 }, + }, + }; + + var html = HtmlRenderer.RenderElements(tree); + + Assert.Contains("data-opacity=\"0.5\"", html); + Assert.DoesNotContain("data-opacity=\"0,5\"", html); + } + finally + { + CultureInfo.CurrentCulture = original; + } + } + + [Fact] + public void RenderElements_FormatsBoundsUsingInvariantCulture() + { + var original = CultureInfo.CurrentCulture; + try + { + CultureInfo.CurrentCulture = new CultureInfo("pt-PT"); // comma decimal + + // Use a fractional scale so the F0 formatter would actually emit a decimal + // separator under a comma-decimal culture if InvariantCulture were not used. + var html = HtmlRenderer.RenderElements(new List + { + new() + { + Id = "e1", + Type = "Label", + Opacity = 0.25, + Bounds = new BoundsInfo { X = 1.5, Y = 2.5, Width = 10.5, Height = 20.5 }, + }, + }, elementScale: 1.5); + + Assert.Contains("position:absolute", html); + Assert.Contains("data-opacity=\"0.25\"", html); + Assert.DoesNotContain("data-opacity=\"0,25\"", html); + } + finally + { + CultureInfo.CurrentCulture = original; + } + } +} diff --git a/src/Cli/Microsoft.Maui.Cli.UnitTests/LocalOriginValidatorTests.cs b/src/Cli/Microsoft.Maui.Cli.UnitTests/LocalOriginValidatorTests.cs new file mode 100644 index 00000000..f8786c79 --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli.UnitTests/LocalOriginValidatorTests.cs @@ -0,0 +1,36 @@ +using Microsoft.Maui.Cli.DevFlow.Inspector; +using Xunit; + +namespace Microsoft.Maui.Cli.UnitTests; + +public class LocalOriginValidatorTests +{ + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("null")] + [InlineData("http://localhost")] + [InlineData("http://localhost:19223")] + [InlineData("http://LOCALHOST:80")] + [InlineData("http://127.0.0.1")] + [InlineData("http://127.0.0.1:5000")] + [InlineData("https://localhost:443")] + public void IsAllowed_ReturnsTrue_ForLoopbackOrAbsentOrigin(string? origin) + { + Assert.True(LocalOriginValidator.IsAllowed(origin)); + } + + [Theory] + [InlineData("http://evil.com")] + [InlineData("https://attacker.example/")] + [InlineData("http://localhost.evil.com")] // not actually localhost + [InlineData("http://127.0.0.1.attacker.com")] + [InlineData("file:///etc/passwd")] + [InlineData("data:text/html,foo")] + [InlineData("not-a-url")] + [InlineData("ftp://localhost")] + public void IsAllowed_ReturnsFalse_ForNonLoopbackOrigin(string origin) + { + Assert.False(LocalOriginValidator.IsAllowed(origin)); + } +} diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs index 93a14857..951a802c 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Broker/BrokerServer.cs @@ -4,6 +4,7 @@ using System.Net.WebSockets; using System.Text; using System.Text.Json.Nodes; +using System.Web; using Microsoft.Maui.Cli.DevFlow.Inspector; namespace Microsoft.Maui.Cli.DevFlow.Broker; @@ -107,6 +108,21 @@ private async Task HandleRequestAsync(HttpListenerContext context) } // HTTP endpoints for CLI + // Block state-mutating endpoints from non-loopback origins BEFORE dispatching + // the handler — otherwise a cross-origin POST to /api/shutdown would still + // tear down the broker even though we return 403. + var origin = context.Request.Headers["Origin"]; + if (method == "POST" && !LocalOriginValidator.IsAllowed(origin)) + { + context.Response.StatusCode = 403; + context.Response.ContentType = "application/json"; + var forbidden = Encoding.UTF8.GetBytes(CliJson.SerializeUntyped(new JsonObject { ["error"] = "Forbidden origin" }, indented: false)); + context.Response.ContentLength64 = forbidden.Length; + await context.Response.OutputStream.WriteAsync(forbidden); + context.Response.Close(); + return; + } + var (statusCode, body) = (method, path) switch { ("GET", "/api/health") => (200, CliJson.SerializeUntyped(new JsonObject @@ -134,7 +150,15 @@ private async Task HandleRequestAsync(HttpListenerContext context) context.Response.StatusCode = statusCode; context.Response.ContentType = "application/json"; - context.Response.Headers.Add("Access-Control-Allow-Origin", "*"); + + // Mirror Origin only for loopback callers; the previous wildcard let any web + // page read /api/agents (leaking IDs) and POST /api/shutdown. + if (LocalOriginValidator.IsAllowed(origin) && !string.IsNullOrEmpty(origin) && origin != "null") + { + context.Response.Headers.Add("Access-Control-Allow-Origin", origin); + context.Response.Headers.Add("Vary", "Origin"); + } + var responseBytes = Encoding.UTF8.GetBytes(body); context.Response.ContentLength64 = responseBytes.Length; await context.Response.OutputStream.WriteAsync(responseBytes); @@ -149,6 +173,16 @@ private async Task HandleRequestAsync(HttpListenerContext context) private async Task HandleAgentWebSocket(HttpListenerContext context) { + // Reject cross-origin WebSocket connections; only the local agent process + // or CLI tools (no Origin header) may register. + var origin = context.Request.Headers["Origin"]; + if (!LocalOriginValidator.IsAllowed(origin)) + { + context.Response.StatusCode = 403; + context.Response.Close(); + return; + } + WebSocketContext wsContext; try { @@ -426,6 +460,11 @@ public void Dispose() _cts?.Cancel(); _idleTimer?.Dispose(); try { _listener?.Close(); } catch { } + foreach (var inspector in _inspectors.Values) + { + try { inspector.Dispose(); } catch { } + } + _inspectors.Clear(); _cts?.Dispose(); } private record AgentConnection(AgentRegistration Registration, WebSocket WebSocket); @@ -508,8 +547,8 @@ private async Task ServeAgentListPage(HttpListenerContext context) { var reg = agent.Registration; sb.AppendLine($"
"); - sb.AppendLine($"{reg.AppName}"); - sb.AppendLine($" — {reg.Platform} ({reg.Tfm}) on port {reg.Port}"); + sb.AppendLine($"{HttpUtility.HtmlEncode(reg.AppName)}"); + sb.AppendLine($" — {HttpUtility.HtmlEncode(reg.Platform)} ({HttpUtility.HtmlEncode(reg.Tfm)}) on port {reg.Port}"); sb.AppendLine($"
"); } } diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs index f83fbbb0..273a0a10 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/HtmlRenderer.cs @@ -1,3 +1,4 @@ +using System.Globalization; using System.Reflection; using System.Text; using System.Web; @@ -13,11 +14,11 @@ namespace Microsoft.Maui.Cli.DevFlow.Inspector; /// public static class HtmlRenderer { - private static string? _templateCache; + private static readonly Lazy _templateCache = new(LoadTemplate, LazyThreadSafetyMode.ExecutionAndPublication); public static string Render(List tree, bool hasScreenshot, int screenshotWidth = 0, int screenshotHeight = 0, double density = 1, double elementScale = 1) { - var template = GetTemplate(); + var template = _templateCache.Value; var (viewportWidth, viewportHeight) = ComputeViewportSize(tree, screenshotWidth, screenshotHeight); // Build the elements HTML (flat list — all elements use window-absolute bounds) @@ -28,12 +29,12 @@ public static string Render(List tree, bool hasScreenshot, int scre ? "\"App" : ""; - // Replace template placeholders + // Replace template placeholders (invariant culture so '.' is the decimal separator) var html = template - .Replace("{{VIEWPORT_WIDTH}}", viewportWidth.ToString("F0")) - .Replace("{{VIEWPORT_HEIGHT}}", viewportHeight.ToString("F0")) - .Replace("{{DENSITY}}", density.ToString("F1")) - .Replace("{{ELEMENT_SCALE}}", elementScale.ToString("F4")) + .Replace("{{VIEWPORT_WIDTH}}", viewportWidth.ToString("F0", CultureInfo.InvariantCulture)) + .Replace("{{VIEWPORT_HEIGHT}}", viewportHeight.ToString("F0", CultureInfo.InvariantCulture)) + .Replace("{{DENSITY}}", density.ToString("F1", CultureInfo.InvariantCulture)) + .Replace("{{ELEMENT_SCALE}}", elementScale.ToString("F4", CultureInfo.InvariantCulture)) .Replace("{{SCREENSHOT}}", screenshotHtml) .Replace("{{ELEMENTS}}", elementsHtml); @@ -65,17 +66,14 @@ private static (double width, double height) ComputeViewportSize(List @@ -101,7 +99,8 @@ private static void RenderSingleElement(StringBuilder sb, ElementInfo element, d if (bounds == null || (bounds.Width <= 0 && bounds.Height <= 0)) return; // Skip elements with no meaningful bounds - var style = $"position:absolute;left:{bounds.X * scale:F0}px;top:{bounds.Y * scale:F0}px;width:{bounds.Width * scale:F0}px;height:{bounds.Height * scale:F0}px;"; + var style = string.Create(CultureInfo.InvariantCulture, + $"position:absolute;left:{bounds.X * scale:F0}px;top:{bounds.Y * scale:F0}px;width:{bounds.Width * scale:F0}px;height:{bounds.Height * scale:F0}px;"); // Build data attributes var attrs = new StringBuilder(); @@ -124,7 +123,7 @@ private static void RenderSingleElement(StringBuilder sb, ElementInfo element, d attrs.Append($" data-isVisible=\"{element.IsVisible.ToString().ToLowerInvariant()}\""); attrs.Append($" data-isEnabled=\"{element.IsEnabled.ToString().ToLowerInvariant()}\""); attrs.Append($" data-isFocused=\"{element.IsFocused.ToString().ToLowerInvariant()}\""); - attrs.Append($" data-opacity=\"{element.Opacity}\""); + attrs.Append(CultureInfo.InvariantCulture, $" data-opacity=\"{element.Opacity:0.###}\""); if (element.Traits is { Count: > 0 }) attrs.Append($" data-traits=\"{Escape(string.Join(",", element.Traits))}\""); diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs index fd860666..ce4d6c04 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/InspectorServer.cs @@ -20,11 +20,19 @@ public sealed class InspectorServer : IDisposable private readonly int _port; private readonly string _agentHost; private readonly int _agentPort; + private readonly AgentClient _client; + private readonly object _cacheLock = new(); + // Lifetime cancellation source; cancelled in Dispose() so broker-mode WS proxies + // (which never call Start() to create _cts) still see shutdown. + private readonly CancellationTokenSource _lifetimeCts = new(); private byte[]? _cachedScreenshot; private DateTime _screenshotCacheTime; private string? _rootPageId; private static readonly TimeSpan ScreenshotCacheDuration = TimeSpan.FromMilliseconds(200); + // Cap request bodies to avoid local DoS via huge POST payloads. + private const long MaxRequestBodyBytes = 1_048_576; // 1 MB + public int Port => _port; public InspectorServer(int port, string agentHost, int agentPort) @@ -32,6 +40,15 @@ public InspectorServer(int port, string agentHost, int agentPort) _port = port; _agentHost = agentHost; _agentPort = agentPort; + _client = new AgentClient(agentHost, agentPort); + } + + private void InvalidateScreenshotCache() + { + lock (_cacheLock) + { + _cachedScreenshot = null; + } } /// @@ -45,16 +62,58 @@ public async Task HandleBrokerRequestAsync(HttpListenerContext context, string p // Handle WebSocket upgrade for /ws/events if (context.Request.IsWebSocketRequest && path.TrimEnd('/') == "/ws/events") { + // Reject cross-origin WebSocket subscriptions (any web page can open a + // WebSocket regardless of same-origin policy — the server must enforce). + var origin = context.Request.Headers["Origin"]; + if (!LocalOriginValidator.IsAllowed(origin)) + { + context.Response.StatusCode = 403; + context.Response.Close(); + return; + } await HandleBrokerWebSocketProxy(context); return; } var method = context.Request.HttpMethod; + + // Mitigate CSRF on state-mutating endpoints: a browser can dispatch a "simple" + // cross-origin POST (text/plain or form-encoded) without a preflight, even + // though it cannot read the response. Reject non-loopback Origins on POST. + if (method == "POST") + { + var origin = context.Request.Headers["Origin"]; + if (!LocalOriginValidator.IsAllowed(origin)) + { + context.Response.StatusCode = 403; + context.Response.Close(); + return; + } + } + string? body = null; if (method == "POST" && context.Request.HasEntityBody) { - using var reader = new System.IO.StreamReader(context.Request.InputStream, context.Request.ContentEncoding); - body = await reader.ReadToEndAsync(); + // Reject oversize bodies to prevent local DoS. + var contentLength = context.Request.ContentLength64; + if (contentLength > MaxRequestBodyBytes) + { + context.Response.StatusCode = 413; + context.Response.Close(); + return; + } + + body = await ReadBoundedBodyAsync( + context.Request.InputStream, + contentLength >= 0 ? contentLength : MaxRequestBodyBytes, + _lifetimeCts.Token); + + if (body == null) + { + context.Response.StatusCode = 413; + context.Response.Close(); + return; + } } var request = new HttpRequestInfo { Method = method, Path = path, Body = body }; @@ -62,7 +121,8 @@ public async Task HandleBrokerRequestAsync(HttpListenerContext context, string p context.Response.StatusCode = statusCode; context.Response.ContentType = contentType; - context.Response.Headers.Add("Access-Control-Allow-Origin", "*"); + // No CORS headers: the inspector UI is served same-origin from the broker. + // Allowing cross-origin would let any web page drive the locally connected app. context.Response.ContentLength64 = responseBody.Length; await context.Response.OutputStream.WriteAsync(responseBody); context.Response.Close(); @@ -81,7 +141,7 @@ public async Task HandleBrokerRequestAsync(HttpListenerContext context, string p } /// - /// Proxies a WebSocket connection from the broker to the agent's /ws/events endpoint. + /// Proxies a WebSocket connection from the broker to the agent's /ws/v1/ui/events endpoint. /// private async Task HandleBrokerWebSocketProxy(HttpListenerContext context) { @@ -89,16 +149,29 @@ private async Task HandleBrokerWebSocketProxy(HttpListenerContext context) var clientWs = wsContext.WebSocket; using var agentWs = new System.Net.WebSockets.ClientWebSocket(); - var agentUri = new Uri($"ws://{_agentHost}:{_agentPort}/ws/events"); + // The agent's WebSocket route is /ws/v1/ui/events (see DevFlowAgentService route map). + var agentUri = new Uri($"ws://{_agentHost}:{_agentPort}/ws/v1/ui/events"); + + // Tie the proxy lifetime to the inspector so Dispose() unblocks ReceiveAsync. + // _lifetimeCts is always non-null (broker mode never calls Start()), and is + // optionally linked to the listener's _cts when running in standalone mode. + using var linkedCts = _cts != null + ? CancellationTokenSource.CreateLinkedTokenSource(_lifetimeCts.Token, _cts.Token) + : CancellationTokenSource.CreateLinkedTokenSource(_lifetimeCts.Token); + var ct = linkedCts.Token; try { - await agentWs.ConnectAsync(agentUri, CancellationToken.None); + await agentWs.ConnectAsync(agentUri, ct); } catch { - await clientWs.CloseAsync(System.Net.WebSockets.WebSocketCloseStatus.EndpointUnavailable, - "Agent not reachable", CancellationToken.None); + try + { + await clientWs.CloseAsync(System.Net.WebSockets.WebSocketCloseStatus.EndpointUnavailable, + "Agent not reachable", CancellationToken.None); + } + catch { } return; } @@ -106,15 +179,16 @@ await clientWs.CloseAsync(System.Net.WebSockets.WebSocketCloseStatus.EndpointUna var buffer = new byte[4096]; try { - while (agentWs.State == System.Net.WebSockets.WebSocketState.Open && + while (!ct.IsCancellationRequested && + agentWs.State == System.Net.WebSockets.WebSocketState.Open && clientWs.State == System.Net.WebSockets.WebSocketState.Open) { - var result = await agentWs.ReceiveAsync(buffer, CancellationToken.None); + var result = await agentWs.ReceiveAsync(buffer, ct); if (result.MessageType == System.Net.WebSockets.WebSocketMessageType.Close) break; await clientWs.SendAsync( new ArraySegment(buffer, 0, result.Count), - result.MessageType, result.EndOfMessage, CancellationToken.None); + result.MessageType, result.EndOfMessage, ct); } } catch { } @@ -145,9 +219,12 @@ public async Task StopAsync() public void Dispose() { + try { _lifetimeCts.Cancel(); } catch { } _cts?.Cancel(); _listener?.Stop(); _cts?.Dispose(); + _lifetimeCts.Dispose(); + _client.Dispose(); } private async Task AcceptLoop(CancellationToken ct) @@ -172,7 +249,15 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) using (client) { var stream = client.GetStream(); - var request = await ReadRequestAsync(stream, ct); + var (request, oversized) = await ReadRequestAsync(stream, ct); + + if (oversized) + { + await WriteResponseAsync(stream, 413, "text/plain", + Encoding.UTF8.GetBytes("Payload Too Large"), ct); + return; + } + if (request == null) return; // Check for WebSocket upgrade on /ws/events @@ -227,16 +312,15 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) private async Task<(int, string, byte[])> HandleRootAsync() { - using var client = new AgentClient(_agentHost, _agentPort); - var tree = await client.GetTreeAsync(); + var tree = await _client.GetTreeAsync(); // Find the root page element (first child of Window with content). // On Mac Catalyst, the default screenshot captures the full screen but element // bounds are relative to the page content. By screenshotting the page element // directly we get a 1:1 match between pixel coordinates and element bounds. var rootPageId = FindRootPageId(tree); - _rootPageId = rootPageId; - var screenshot = await GetCachedScreenshotAsync(client, rootPageId); + lock (_cacheLock) { _rootPageId = rootPageId; } + var screenshot = await GetCachedScreenshotAsync(rootPageId); var hasScreenshot = screenshot?.Length > 0; double viewportWidth = 800, viewportHeight = 600; @@ -257,13 +341,15 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) /// private async Task<(int, string, byte[])> HandleStateAsync() { - using var client = new AgentClient(_agentHost, _agentPort); - var tree = await client.GetTreeAsync(); + var tree = await _client.GetTreeAsync(); var rootPageId = FindRootPageId(tree); - _rootPageId = rootPageId; - _cachedScreenshot = null; // force fresh screenshot - var screenshot = await GetCachedScreenshotAsync(client, rootPageId); + lock (_cacheLock) + { + _rootPageId = rootPageId; + _cachedScreenshot = null; // force fresh screenshot + } + var screenshot = await GetCachedScreenshotAsync(rootPageId); var hasScreenshot = screenshot?.Length > 0; double viewportWidth = 800, viewportHeight = 600; @@ -302,19 +388,24 @@ private async Task HandleClientAsync(TcpClient client, CancellationToken ct) return window.Children[^1].Id; } - /// Reads width/height from PNG IHDR chunk (bytes 16-23). + /// Reads width/height from PNG IHDR chunk (bytes 16-23) after validating PNG signature. private static (int width, int height) GetPngDimensions(byte[] png) { - if (png.Length < 24) return (0, 0); + // PNG magic: 137 80 78 71 13 10 26 10 + ReadOnlySpan pngSig = [137, 80, 78, 71, 13, 10, 26, 10]; + if (png.Length < 24 || !png.AsSpan(0, 8).SequenceEqual(pngSig)) + return (0, 0); int w = (png[16] << 24) | (png[17] << 16) | (png[18] << 8) | png[19]; int h = (png[20] << 24) | (png[21] << 16) | (png[22] << 8) | png[23]; + if (w < 0 || h < 0) return (0, 0); return (w, h); } private async Task<(int, string, byte[])> HandleScreenshotAsync() { - using var client = new AgentClient(_agentHost, _agentPort); - var png = await GetCachedScreenshotAsync(client, _rootPageId); + string? rootPageId; + lock (_cacheLock) { rootPageId = _rootPageId; } + var png = await GetCachedScreenshotAsync(rootPageId); if (png == null || png.Length == 0) return (404, "text/plain", Encoding.UTF8.GetBytes("No screenshot available")); return (200, "image/png", png); @@ -333,14 +424,21 @@ private static (int width, int height) GetPngDimensions(byte[] png) return (200, contentType, ms.ToArray()); } - private async Task GetCachedScreenshotAsync(AgentClient client, string? elementId = null) + private async Task GetCachedScreenshotAsync(string? elementId = null) { - if (_cachedScreenshot != null && DateTime.UtcNow - _screenshotCacheTime < ScreenshotCacheDuration) - return _cachedScreenshot; + lock (_cacheLock) + { + if (_cachedScreenshot != null && DateTime.UtcNow - _screenshotCacheTime < ScreenshotCacheDuration) + return _cachedScreenshot; + } - _cachedScreenshot = await client.ScreenshotAsync(elementId: elementId); - _screenshotCacheTime = DateTime.UtcNow; - return _cachedScreenshot; + var fresh = await _client.ScreenshotAsync(elementId: elementId); + lock (_cacheLock) + { + _cachedScreenshot = fresh; + _screenshotCacheTime = DateTime.UtcNow; + } + return fresh; } // ── Proxy handlers ── @@ -359,8 +457,7 @@ private static (int width, int height) GetPngDimensions(byte[] png) var x = xProp.GetDouble(); var y = yProp.GetDouble(); - using var client = new AgentClient(_agentHost, _agentPort); - var hitResult = await client.HitTestAsync(x, y); + var hitResult = await _client.HitTestAsync(x, y); // Parse hit-test result — response is { elements: [{ id, ... }, ...] } using var hitDoc = JsonDocument.Parse(hitResult); @@ -373,10 +470,10 @@ private static (int width, int height) GetPngDimensions(byte[] png) var elementId = elements[i].GetProperty("id").GetString(); if (!string.IsNullOrEmpty(elementId)) { - var success = await client.TapAsync(elementId); + var success = await _client.TapAsync(elementId); if (success) { - _cachedScreenshot = null; + InvalidateScreenshotCache(); return (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")); } } @@ -391,9 +488,8 @@ private static (int width, int height) GetPngDimensions(byte[] png) var elementId = elIdProp.GetString(); if (!string.IsNullOrEmpty(elementId)) { - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.TapAsync(elementId); - _cachedScreenshot = null; + var success = await _client.TapAsync(elementId); + InvalidateScreenshotCache(); return success ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); @@ -417,8 +513,7 @@ private static (int width, int height) GetPngDimensions(byte[] png) // If coordinates provided, hit-test and try each element for scroll if (root.TryGetProperty("x", out var xProp) && root.TryGetProperty("y", out var yProp)) { - using var client = new AgentClient(_agentHost, _agentPort); - var hitResult = await client.HitTestAsync(xProp.GetDouble(), yProp.GetDouble()); + var hitResult = await _client.HitTestAsync(xProp.GetDouble(), yProp.GetDouble()); using var hitDoc = JsonDocument.Parse(hitResult); var elements = hitDoc.RootElement.GetProperty("elements"); @@ -428,10 +523,10 @@ private static (int width, int height) GetPngDimensions(byte[] png) var elementId = elements[i].GetProperty("id").GetString(); if (!string.IsNullOrEmpty(elementId)) { - var success = await client.ScrollAsync(elementId: elementId, deltaX: deltaX, deltaY: deltaY); + var success = await _client.ScrollAsync(elementId: elementId, deltaX: deltaX, deltaY: deltaY); if (success) { - _cachedScreenshot = null; + InvalidateScreenshotCache(); return (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")); } } @@ -440,9 +535,8 @@ private static (int width, int height) GetPngDimensions(byte[] png) // Fallback: scroll without element target { - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.ScrollAsync(deltaX: deltaX, deltaY: deltaY); - _cachedScreenshot = null; + var success = await _client.ScrollAsync(deltaX: deltaX, deltaY: deltaY); + InvalidateScreenshotCache(); return success ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); @@ -471,9 +565,8 @@ private static (int width, int height) GetPngDimensions(byte[] png) var distance = Math.Sqrt(dx * dx + dy * dy); - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.GestureAsync("swipe", direction: direction, distance: distance); - _cachedScreenshot = null; + var success = await _client.GestureAsync("swipe", direction: direction, distance: distance); + InvalidateScreenshotCache(); return success ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); @@ -484,9 +577,8 @@ private static (int width, int height) GetPngDimensions(byte[] png) private async Task<(int, string, byte[])> HandleProxyBackAsync() { - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.BackAsync(); - _cachedScreenshot = null; + var success = await _client.BackAsync(); + InvalidateScreenshotCache(); return success ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); @@ -506,9 +598,8 @@ private static (int width, int height) GetPngDimensions(byte[] png) if (string.IsNullOrEmpty(elementId) || text == null) return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"elementId and text required\"}")); - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.FillAsync(elementId, text); - _cachedScreenshot = null; + var success = await _client.FillAsync(elementId, text); + InvalidateScreenshotCache(); return success ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); @@ -528,9 +619,8 @@ private static (int width, int height) GetPngDimensions(byte[] png) if (string.IsNullOrEmpty(key)) return (400, "application/json", Encoding.UTF8.GetBytes("{\"error\":\"key required\"}")); - using var client = new AgentClient(_agentHost, _agentPort); - var success = await client.KeyAsync(key, elementId); - _cachedScreenshot = null; + var success = await _client.KeyAsync(key, elementId); + InvalidateScreenshotCache(); return success ? (200, "application/json", Encoding.UTF8.GetBytes("{\"ok\":true}")) : (500, "application/json", Encoding.UTF8.GetBytes("{\"ok\":false}")); @@ -606,7 +696,34 @@ private static async Task SendWebSocketFrameAsync(NetworkStream stream, byte[] p // ── HTTP parsing helpers ── - private static async Task ReadRequestAsync(NetworkStream stream, CancellationToken ct) + /// + /// Reads a request body from a stream up to , decoding as UTF-8. + /// Returns null if the body exceeds the cap. Decoding once at the end avoids splitting + /// multi-byte UTF-8 sequences across chunk reads. A per-read timeout prevents slow-drip + /// clients from holding the handler open. + /// + private static async Task ReadBoundedBodyAsync(Stream input, long maxBytes, CancellationToken ct = default) + { + using var ms = new MemoryStream(); + var buffer = new byte[8192]; + long total = 0; + while (true) + { + using var perReadCts = CancellationTokenSource.CreateLinkedTokenSource(ct); + perReadCts.CancelAfter(TimeSpan.FromSeconds(10)); + int read; + try { read = await input.ReadAsync(buffer.AsMemory(), perReadCts.Token); } + catch { return null; } + if (read <= 0) break; + total += read; + if (total > maxBytes) + return null; + ms.Write(buffer, 0, read); + } + return Encoding.UTF8.GetString(ms.GetBuffer(), 0, (int)ms.Length); + } + + private static async Task<(HttpRequestInfo? Request, bool Oversized)> ReadRequestAsync(NetworkStream stream, CancellationToken ct) { var buffer = new byte[8192]; int read; @@ -616,22 +733,23 @@ private static async Task SendWebSocketFrameAsync(NetworkStream stream, byte[] p try { read = await stream.ReadAsync(buffer, timeoutCts.Token); - if (read == 0) return null; + if (read == 0) return (null, false); } - catch { return null; } + catch { return (null, false); } - var raw = Encoding.UTF8.GetString(buffer, 0, read); + // Parse headers from the ASCII portion. The body is read as raw bytes below + // to avoid splitting multi-byte UTF-8 sequences across buffer boundaries. + var raw = Encoding.ASCII.GetString(buffer, 0, read); var headerEnd = raw.IndexOf("\r\n\r\n", StringComparison.Ordinal); - if (headerEnd < 0) return null; + if (headerEnd < 0) return (null, false); var headerSection = raw[..headerEnd]; - var bodySection = raw[(headerEnd + 4)..]; var lines = headerSection.Split("\r\n"); - if (lines.Length == 0) return null; + if (lines.Length == 0) return (null, false); var requestLine = lines[0].Split(' '); - if (requestLine.Length < 2) return null; + if (requestLine.Length < 2) return (null, false); var method = requestLine[0].ToUpperInvariant(); var path = requestLine[1].Split('?')[0].TrimEnd('/'); @@ -649,25 +767,44 @@ private static async Task SendWebSocketFrameAsync(NetworkStream stream, byte[] p } } - // Read remaining body if Content-Length indicates more - var body = bodySection; - if (headers.TryGetValue("content-length", out var clStr) && int.TryParse(clStr, out var contentLength)) + // Read body as raw bytes, then decode as UTF-8 once. + string? body = null; + if (headers.TryGetValue("content-length", out var clStr) && int.TryParse(clStr, out var contentLength) && contentLength > 0) { - while (Encoding.UTF8.GetByteCount(body) < contentLength) + if (contentLength > MaxRequestBodyBytes) + return (null, true); + + var bodyStart = headerEnd + 4; + var bytesAlreadyRead = read - bodyStart; + var bodyBytes = new byte[contentLength]; + + if (bytesAlreadyRead > 0) { - var extraRead = await stream.ReadAsync(buffer, ct); - if (extraRead == 0) break; - body += Encoding.UTF8.GetString(buffer, 0, extraRead); + var copy = Math.Min(bytesAlreadyRead, contentLength); + Buffer.BlockCopy(buffer, bodyStart, bodyBytes, 0, copy); } + + int totalBodyRead = Math.Min(Math.Max(0, bytesAlreadyRead), contentLength); + while (totalBodyRead < contentLength) + { + using var perReadCts = CancellationTokenSource.CreateLinkedTokenSource(ct); + perReadCts.CancelAfter(TimeSpan.FromSeconds(10)); + int extra; + try { extra = await stream.ReadAsync(bodyBytes.AsMemory(totalBodyRead, contentLength - totalBodyRead), perReadCts.Token); } + catch { return (null, false); } + if (extra == 0) break; + totalBodyRead += extra; + } + body = Encoding.UTF8.GetString(bodyBytes, 0, totalBodyRead); } - return new HttpRequestInfo + return (new HttpRequestInfo { Method = method, Path = path, Headers = headers, Body = body - }; + }, false); } private static async Task WriteResponseAsync(NetworkStream stream, int statusCode, string contentType, byte[] body, CancellationToken ct) @@ -678,16 +815,16 @@ private static async Task WriteResponseAsync(NetworkStream stream, int statusCod 400 => "Bad Request", 404 => "Not Found", 405 => "Method Not Allowed", + 413 => "Payload Too Large", 500 => "Internal Server Error", _ => "Unknown" }; + // No CORS headers: the inspector UI is served same-origin; allowing + // cross-origin would let any web page drive the locally connected app. var header = $"HTTP/1.1 {statusCode} {statusText}\r\n" + $"Content-Type: {contentType}\r\n" + $"Content-Length: {body.Length}\r\n" + - "Access-Control-Allow-Origin: *\r\n" + - "Access-Control-Allow-Methods: GET, POST, OPTIONS\r\n" + - "Access-Control-Allow-Headers: Content-Type\r\n" + "Connection: close\r\n\r\n"; await stream.WriteAsync(Encoding.UTF8.GetBytes(header), ct); diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/LocalOriginValidator.cs b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/LocalOriginValidator.cs new file mode 100644 index 00000000..0e660094 --- /dev/null +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/LocalOriginValidator.cs @@ -0,0 +1,33 @@ +namespace Microsoft.Maui.Cli.DevFlow.Inspector; + +/// +/// Origin-header validation for localhost-only HTTP/WebSocket endpoints. +/// Used to defend against cross-origin attacks (CSRF on POST endpoints, hijacked +/// WebSocket subscriptions) when the only legitimate callers are the DevFlow CLI +/// (no Origin header) or a browser session on the same loopback origin. +/// +internal static class LocalOriginValidator +{ + /// + /// Returns true if the origin is either absent (non-browser client) or a + /// loopback HTTP/HTTPS URI (http://localhost*, http://127.0.0.1*, http://[::1]*). + /// + public static bool IsAllowed(string? origin) + { + // No Origin header: non-browser client (e.g. CLI tool, curl). Permit. + if (string.IsNullOrEmpty(origin) || origin == "null") + return true; + + if (!Uri.TryCreate(origin, UriKind.Absolute, out var uri)) + return false; + + if (uri.Scheme != Uri.UriSchemeHttp && uri.Scheme != Uri.UriSchemeHttps) + return false; + + var host = uri.Host; + return host.Equals("localhost", StringComparison.OrdinalIgnoreCase) + || host == "127.0.0.1" + || host == "[::1]" + || host == "::1"; + } +} diff --git a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs index 9a7fe737..6a832316 100644 --- a/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs +++ b/src/DevFlow/Microsoft.Maui.DevFlow.Inspector.Tests/InspectorPageTests.cs @@ -1,3 +1,4 @@ +using System.Globalization; using Microsoft.Playwright; using Xunit; @@ -8,6 +9,9 @@ namespace Microsoft.Maui.DevFlow.Inspector.Tests; /// Requires the broker running with a connected MAUI app. /// The inspector is available at http://localhost:19223/inspector/. /// Set INSPECTOR_URL environment variable to override the default URL. +/// +/// The default URL points to the broker's single-agent fallback route +/// (the broker uses the only connected agent when the id segment doesn't match). /// [Collection("Inspector")] public class InspectorPageTests : IAsyncLifetime @@ -17,7 +21,11 @@ public class InspectorPageTests : IAsyncLifetime private IBrowserContext _context = null!; private IPage _page = null!; - private string BaseUrl => Environment.GetEnvironmentVariable("INSPECTOR_URL") ?? "http://localhost:19223/inspector/"; + private Uri BaseUri => new(Environment.GetEnvironmentVariable("INSPECTOR_URL") ?? "http://localhost:19223/inspector/default/"); + + private string BaseUrl => BaseUri.ToString(); + + private string ResolveUrl(string relativePath) => new Uri(BaseUri, relativePath).ToString(); public async Task InitializeAsync() { @@ -45,8 +53,8 @@ public async Task ViewportUsesWindowDimensionsFromAgent() var height = await viewport.GetAttributeAsync("data-height"); // Window dimensions should be positive and NOT the old iPhone defaults - var w = double.Parse(width!); - var h = double.Parse(height!); + var w = double.Parse(width!, CultureInfo.InvariantCulture); + var h = double.Parse(height!, CultureInfo.InvariantCulture); Assert.True(w > 0, "Viewport width should be positive"); Assert.True(h > 0, "Viewport height should be positive"); Assert.NotEqual(390, w); // Not hardcoded iPhone width @@ -150,7 +158,7 @@ public async Task DataAttributesUseCamelCase() [Fact] public async Task CssServedSeparately() { - var response = await _page.APIRequest.GetAsync($"{BaseUrl}/devflow.css"); + var response = await _page.APIRequest.GetAsync(ResolveUrl("devflow.css")); Assert.True(response.Ok); var text = await response.TextAsync(); Assert.Contains("#app-viewport", text); @@ -239,7 +247,7 @@ await _page.RouteAsync("**/api/tap", async route => [Fact] public async Task ScreenshotEndpointReturnsPng() { - var response = await _page.APIRequest.GetAsync($"{BaseUrl}/screenshot.png"); + var response = await _page.APIRequest.GetAsync(ResolveUrl("screenshot.png")); Assert.True(response.Ok); var body = await response.BodyAsync(); @@ -253,7 +261,7 @@ public async Task ScreenshotEndpointReturnsPng() [Fact] public async Task StateEndpointReturnsJsonWithElements() { - var response = await _page.APIRequest.GetAsync($"{BaseUrl}api/state"); + var response = await _page.APIRequest.GetAsync(ResolveUrl("api/state")); Assert.True(response.Ok); var text = await response.TextAsync(); var json = System.Text.Json.JsonDocument.Parse(text); From 1be051a772d2550ac4326f8d8fb08063e90d28d7 Mon Sep 17 00:00:00 2001 From: Rui Marinho Date: Tue, 2 Jun 2026 14:04:47 +0100 Subject: [PATCH 08/13] feat(inspector): inline editor for text input elements Clicking an Entry, Editor, SearchBar, or other text-input element in the web inspector now opens a teal-bordered overlay editor (input or textarea) on top of the field, pre-filled with the field's current text. Pressing Enter or clicking away commits the change via POST /api/fill; Escape cancels. The native control still receives a tap so it focuses on the app side. Detection uses data-type (Entry, Editor, SearchBar, TextField, TextBox, TextArea, TextView, UITextField, UITextView, EditText, NSTextField) and a heuristic on data-traits for 'textinput' / 'editable'. Uses document.elementFromPoint(clientX, clientY) to find the underlying element because the existing gesture handler calls viewport.setPointerCapture(), which causes real mouse clicks to report the viewport itself as e.target. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../DevFlow/Inspector/Web/devflow.js | 109 +++++++++++++++++- 1 file changed, 108 insertions(+), 1 deletion(-) diff --git a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js index 728a1d21..b1a13869 100644 --- a/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js +++ b/src/Cli/Microsoft.Maui.Cli/DevFlow/Inspector/Web/devflow.js @@ -143,9 +143,116 @@ }, delayMs || 300); } - // ── Click → Tap ── + // ── Click → Tap (with text-input awareness) ── + // Element types that should open a text editor instead of just tapping. + const TEXT_INPUT_TYPES = new Set([ + 'Entry', 'Editor', 'SearchBar', 'SearchHandler', + 'TextField', 'TextBox', 'TextArea', 'TextView', + 'UITextField', 'UITextView', + 'EditText', 'NSTextField', + ]); + + function isTextInput(el) { + if (!el || !el.classList || !el.classList.contains('devflow-element')) return false; + const type = el.dataset.type || ''; + if (TEXT_INPUT_TYPES.has(type)) return true; + // Heuristic: traits often expose "TextInput" / "Editable" + const traits = (el.dataset.traits || '').toLowerCase(); + return traits.includes('textinput') || traits.includes('editable'); + } + + // Overlay editor that we float on top of the clicked text element. + let activeEditor = null; + function closeEditor(commit) { + if (!activeEditor) return; + const editor = activeEditor; + activeEditor = null; + if (commit) { + const elementId = editor.dataset.elementId; + const text = editor.value; + fetch(`${basePath}/api/fill`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ elementId, text }), + }).then(() => scheduleRefresh(300)).catch(err => console.error('Fill failed:', err)); + } + editor.remove(); + } + + function openEditor(targetEl) { + closeEditor(false); + const elementId = targetEl.getAttribute('data-id'); + if (!elementId) return; + + const rect = targetEl.getBoundingClientRect(); + const vpRect = viewport.getBoundingClientRect(); + const isMultiline = ['Editor', 'TextArea', 'TextView', 'UITextView'].includes(targetEl.dataset.type || ''); + const editor = document.createElement(isMultiline ? 'textarea' : 'input'); + if (!isMultiline) editor.type = 'text'; + editor.value = targetEl.dataset.text || targetEl.dataset.value || ''; + editor.dataset.elementId = elementId; + Object.assign(editor.style, { + position: 'absolute', + left: (rect.left - vpRect.left) + 'px', + top: (rect.top - vpRect.top) + 'px', + width: rect.width + 'px', + height: rect.height + 'px', + zIndex: '10000', + background: 'rgba(255,255,255,0.97)', + color: '#000', + border: '2px solid #4ec9b0', + borderRadius: '2px', + padding: '2px 4px', + font: 'inherit', + fontSize: Math.max(11, Math.min(20, rect.height * 0.5)) + 'px', + outline: 'none', + boxSizing: 'border-box', + resize: 'none', + }); + + editor.addEventListener('keydown', (ev) => { + if (ev.key === 'Escape') { + ev.preventDefault(); + closeEditor(false); + } else if (ev.key === 'Enter' && !isMultiline) { + ev.preventDefault(); + closeEditor(true); + } + }); + editor.addEventListener('blur', () => closeEditor(true)); + + viewport.appendChild(editor); + activeEditor = editor; + // Use a microtask so the click that opened us doesn't immediately blur it. + setTimeout(() => { editor.focus(); editor.select(); }, 0); + } + viewport.addEventListener('click', async (e) => { if (isDragging) return; + // If the user clicks back into the active editor, ignore. + if (activeEditor && (e.target === activeEditor || activeEditor.contains(e.target))) return; + + // setPointerCapture(viewport) makes e.target be the viewport itself for real + // mouse clicks, so use elementFromPoint to find the actual element under the + // cursor. Temporarily hide any active editor so it doesn't shadow the click. + let underCursor = document.elementFromPoint(e.clientX, e.clientY); + if (underCursor === viewport || underCursor === screenshot) { + // Both are pointer-events:none / non-interactive overlays; fall back to e.target. + underCursor = e.target; + } + let textEl = underCursor; + while (textEl && textEl !== viewport && !isTextInput(textEl)) textEl = textEl.parentElement; + if (textEl && textEl !== viewport && isTextInput(textEl)) { + // Still send a tap so the native control gets focus on the app side. + const { x: tx, y: ty } = toAppCoords(e.clientX, e.clientY); + fetch(`${basePath}/api/tap`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ x: tx, y: ty }), + }).catch(err => console.error('Tap failed:', err)); + openEditor(textEl); + return; + } const { x, y } = toAppCoords(e.clientX, e.clientY); From 60ef921be23541930bccdc6a1b50cf094ab8b742 Mon Sep 17 00:00:00 2001 From: Rui Marinho Date: Tue, 2 Jun 2026 14:19:23 +0100 Subject: [PATCH 09/13] docs(inspector): clarify implemented vs future-work sections The inspector spec previously documented a standalone `maui devflow inspector` command on port 5223 with a toolbar and nested element tree. The implemented behavior is broker-hosted at port 19223 with a flat element list and no toolbar. Update the architecture diagram and Usage section to reflect the current broker-hosted setup at http://localhost:19223/inspector/, and re-label the standalone command as future work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/DevFlow/inspector.md | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/docs/DevFlow/inspector.md b/docs/DevFlow/inspector.md index 004b698e..4cebff81 100644 --- a/docs/DevFlow/inspector.md +++ b/docs/DevFlow/inspector.md @@ -1,6 +1,6 @@ # DevFlow Web Inspector -> **Status**: Design spec — V1 implementation in progress +> **Status**: Mixed — the broker-hosted inspector at `http://localhost:19223/inspector/` is implemented and shipping in `maui devflow broker`. Sections that describe a standalone `maui devflow inspector` command, a `