diff --git a/apps/web/src/artifacts/artifactRef.test.ts b/apps/web/src/artifacts/artifactRef.test.ts index e3d2449c5..154242eaa 100644 --- a/apps/web/src/artifacts/artifactRef.test.ts +++ b/apps/web/src/artifacts/artifactRef.test.ts @@ -50,6 +50,8 @@ describe("artifactRefFromProps", () => { expect(artifactRefFromProps(undefined)).toBeNull(); // An unknown kind is dropped rather than echoed into the chip. expect(artifactRefFromProps({ artifact_id: "a", version: 1, kind: "x" })?.kind).toBe(""); + // A chart (ADR 0116) is a known kind, so an untitled one still says what it is. + expect(artifactRefFromProps({ artifact_id: "a", version: 1, kind: "vega-lite" })?.kind).toBe("vega-lite"); }); it("names an untitled artifact by its kind", () => { diff --git a/apps/web/src/artifacts/artifactRef.ts b/apps/web/src/artifacts/artifactRef.ts index 7b7f644d8..8eaba29fc 100644 --- a/apps/web/src/artifacts/artifactRef.ts +++ b/apps/web/src/artifacts/artifactRef.ts @@ -15,7 +15,7 @@ export const ARTIFACT_REF_COMPONENT = "artifact-ref"; /** The artifact plugin's panel — `plugin::` from its manifest. */ export const ARTIFACT_VIEW_KEY = "plugin:artifact:artifact"; -const KINDS = new Set(["html", "svg", "mermaid", "react", "markdown", "file"]); +const KINDS = new Set(["html", "svg", "mermaid", "react", "markdown", "vega-lite", "file"]); export type ArtifactRef = { id: string; version: number; title: string; kind: string }; diff --git a/changelog.d/4025.added.md b/changelog.d/4025.added.md new file mode 100644 index 000000000..999270673 --- /dev/null +++ b/changelog.d/4025.added.md @@ -0,0 +1,9 @@ +- **Charts as Vega-Lite specs, and a plugin-services seam, for the local-first Data Analyst (ADR 0116) (#4025).** + The Artifact panel adds a `vega-lite` kind: a small JSON spec with its rows inline is drawn by vendored, SRI-pinned + Vega / Vega-Lite / vega-embed, in the console's own chart palette (dark or light, redrawn on a theme switch). It + runs under a no-network CSP: no `eval`, a loader that refuses every fetch, and inline data only. A chart is a query + plus a few hundred bytes of spec instead of a hand-written component. In the end-to-end run, "what were my best + weekdays last quarter? chart it" went from send to chart in about 18 s, against about 50 s for the component path. + Plugins also gain a call seam: `registry.register_service(name, fn)` offers a callable, and + `graph.sdk.service(".")` resolves it, or returns `None` when the provider is off. The first service + is `artifact.show`, which the new Data Analyst plugin (`protoLabsAI/data-plugin`) uses to put its charts in the panel. diff --git a/docs/adr/0116-local-first-data-analyst-duckdb-and-vega-lite-charts.md b/docs/adr/0116-local-first-data-analyst-duckdb-and-vega-lite-charts.md new file mode 100644 index 000000000..23f335bed --- /dev/null +++ b/docs/adr/0116-local-first-data-analyst-duckdb-and-vega-lite-charts.md @@ -0,0 +1,170 @@ +# 0116 — A local-first data analyst: DuckDB over files in place, charts as Vega-Lite specs, and a plugin-service call seam + +- Status: Proposed +- Date: 2026-10-03 +- Builds on: [ADR 0038](./0038-generative-ui-artifacts-two-mode.md) (the Artifact panel and its sandboxed frame), [ADR 0092](./0092-desktop-document-baseline-and-versioned-file-artifacts.md) (file artifacts), [ADR 0039](./0039-plugin-event-bus.md) (plugins talk without importing each other), [ADR 0043](./0043-plugin-consumption-sdk-workflows-extraction.md) (the consumption SDK), [ADR 0019](./0019-plugin-config-settings-secrets.md) §3b (`spawns: true`, the operator-only setting fence). +- Refs: the `chart my week` launch demo (~50 s from send to chart); #4019 (the vendoring + nonce-CSP pattern this reuses). + +## Context + +The Data Analyst is a local-first agent shape: *your agent, your data, your way*. The operator points it at data that already sits on their disk — CSV, Parquet, JSON, Excel and SQLite files, or a folder of them — and the agent explores it, answers questions in SQL, and puts charts in the console. Nothing is uploaded, and nothing is imported into a database first. + +Three things are missing today. + +**1. There is no data engine.** The agent can read a file with `read_file`, or write Python with `execute_code`. Neither is a good way to answer "what were my best weekdays last quarter?" over 100k rows. The first floods the context window. The second makes the model author a pandas program for every question and is only as safe as the code it writes. + +**2. Charts are slow because the model draws them by hand.** The only way to put a chart in the panel is a `react` (or `html`) artifact. The model writes a whole component: imports, layout, axis formatting, colours, and the data itself typed out as a JavaScript literal. In the launch demo, `chart my week` took about 50 s from send to chart, and almost all of that time was the model generating that component. A chart is not a component. It is a *query* plus a *mapping* from columns to marks. Vega-Lite expresses that mapping in a few hundred bytes of JSON. + +**3. A plugin cannot get a result back from another plugin.** The data plugin has to create an artifact and report which one it made. Plugins coordinate over the event bus (ADR 0039), which is fire-and-forget: no return value, and no way to ask "is that plugin even on?". The only other route is to import the artifact plugin's internals (`plugins.artifact._tools._show`). That is the coupling the plugin contract exists to prevent: it breaks on any refactor, and it fails at import time when the plugin is disabled. No plugin creates artifacts today (checked at write time: the only cross-module reader is `server/chat_session_ops.py`, which imports `resolve_for_bundle` defensively). So no existing seam can be reused. + +## Decision drivers + +- **Time to chart.** The model should write the least it can (a SQL query and a small spec) and the host should do the rest. +- **Read-only, enforced by the engine.** The agent's SQL must not be able to write, attach, install, load, or read anything outside what the operator allowed. A regex over SQL text is not a fence. +- **Operator-owned scope.** Only the operator decides which folders the agent can read. The agent cannot widen that scope itself, even with `set_config`. +- **Offline and same-origin.** Charts render with no network, like mermaid and the pptx renderer (#4019). +- **The console's look.** A chart should look like the console it sits in, in both dark and light themes, without the model choosing any colours. +- **Plugins stay decoupled.** A plugin must never import another plugin's internals. + +## Options considered + +### Engine + +| Option | For | Against | +|---|---|---| +| **DuckDB, embedded (chosen)** | Queries CSV, Parquet and JSON *in place*, with no import step. It is columnar and fast on a laptop, and installs as one MIT-licensed wheel with no server. It has engine-level lockdown settings: `enable_external_access`, `allowed_paths`, `lock_configuration`. | It cannot read SQLite or Excel without extensions that `INSTALL` from the network (D2). | +| SQLite (stdlib) | Already present. | Every file has to be imported first, its analytics functions are weak, and it has no file-scanning lockdown. | +| pandas / polars in `execute_code` | Flexible. | The model writes a program for every question, and the only fence is the sandbox. Neither library is a host dependency. | + +### Chart format + +| Option | For | Against | +|---|---|---| +| **Vega-Lite spec + inline rows (chosen)** | A declarative grammar of graphics: a chart is about 300 bytes of JSON that models already write well. Vega can be themed from tokens, has tooltips, and runs offline with no `eval` (D5). | About 830 KB of vendored JS. | +| React + chart.js (status quo) | Already vendored. | It is the slow path that motivated this ADR: the model writes the component and the data. | +| ECharts / Plotly specs | Popular. | Plotly is about 3.5 MB. Both are less constrained, and Plotly can load remote resources unless configured carefully. Models write them less reliably. | + +### Cross-plugin seam + +| Option | Against | +|---|---| +| Import `plugins.artifact` internals | It is the coupling the plugin contract forbids. It breaks on any refactor and on a disabled plugin. | +| Event bus request/reply | The bus is fire-and-forget by design (ADR 0039). A reply would need correlation IDs, timeouts, and a second topic, which reinvents an RPC badly. | +| A dedicated `sdk.show_artifact()` in core | It would make `graph/` depend on one plugin, and every later plugin-to-plugin need would want its own SDK function. | +| **Plugin services (chosen)** | — a generic, small, named-callable registry (D6). | + +## Decision + +### D1 — It ships as a plugin first: `data` (Data Analyst), in its own repo + +The analyst is `protoLabsAI/data-plugin` (plugin id `data`, MIT, off by default), with the same conventions as `campaign-plugin`: a host-free test suite, CI, and the shared release workflow. It has seven tools: + +| Tool | Does | +|---|---| +| `data_connect(path, name)` | Registers a file, or a folder of files (bounded walk), as named sources. SQLite gives one source per table and Excel one per sheet. | +| `data_sources()` | Lists sources with their kind, path, row count and column count. | +| `data_schema(source)` | Shows columns, types and sample rows. | +| `data_query(sql)` | Runs a read-only SELECT and returns a compact table, with row and time caps. | +| `data_profile(source)` | Reports per-column nulls, cardinality, min/max/mean/stddev, top values, and IQR outliers. | +| `data_chart(sql, vega_lite_spec, title)` | Runs the query, inlines the rows into the spec, and hands it to the Artifact panel (D5, D6). | +| `data_export(sql, format)` | Writes CSV or Parquet into the agent's workspace (D7). | + +It also ships two skills: `exploring-a-dataset` and `building-a-chart` (the spec patterns that read well). A "Data Analyst" archetype bundle (catalog row, SOUL, default skills) comes later, and its scope is an open question. + +### D2 — DuckDB, embedded, queries files where they are + +Each source is a DuckDB **view** over its file (`read_csv` / `read_parquet` / `read_json_auto`), so a query reads the operator's file directly and nothing is copied. The wheel has only `core_functions`, `icu`, `json` and `parquet` built in. Reading SQLite or Excel would need `INSTALL sqlite`/`excel`, which is a network fetch of native code, and D3 refuses that by design. So those two formats are **snapshotted to Parquet** in the plugin's private cache: + +- SQLite through the stdlib `sqlite3`, opened read-only. +- Excel through `openpyxl`, an optional MIT dependency. + +The snapshot is taken on a private connection that the agent's SQL never reaches. It is re-taken when the source's mtime or size changes, and that check runs at query time. + +### D3 — Read-only is enforced by the engine, and the SQL text is checked as a second layer + +**Every query gets a fresh in-memory connection, configured in this order:** + +1. `autoinstall_known_extensions=false` and `autoload_known_extensions=false`, set at connect time. +2. A private `temp_directory`. DuckDB adds its temp directory to `allowed_directories` automatically, and the default temp directory is relative to the process working directory. +3. `memory_limit` and `threads`. +4. `allowed_paths` set to the **exact resolved files** of the currently valid sources, plus their Parquet snapshots. This setting cannot be passed at connect time ("before the database is started"), so it is a `SET` here. +5. `enable_external_access=false`. +6. One view per source. +7. `lock_configuration=true`. + +The result was verified against DuckDB 1.5.6. Under this configuration the engine itself refuses all of the following: + +- reads outside the allowlist, including a sibling `.env` via `read_text` and `glob()` +- `COPY … TO`, including onto an allowed path +- `ATTACH` of a file +- `EXPORT DATABASE` +- `INSTALL` and `LOAD` +- `SET` and `RESET` of any option +- `https://` URLs + +`getenv()` does not exist in the embedded build. + +**Statement guard (defence in depth).** `conn.extract_statements(sql)` must return exactly **one** statement of type `SELECT`. This closes what the engine still allows: an in-memory `ATTACH ':memory:'`, `CREATE TABLE`, and `PRAGMA`. It also refuses multi-statement batches. + +**Caps.** +- Rows: `fetchmany(cap + 1)`, with truncation reported to the model. +- Wall time: a timer calls `conn.interrupt()`. +- Memory: `memory_limit`. + +The defaults are 200 rows, 20 s and 1 GB, and they are plugin settings. + +### D4 — The scope is the operator's `data_dirs`, an operator-only allowlist + +`data_dirs` is a plugin setting marked **`spawns: true`**, the same marker campaign uses for `upload_dirs`. It is core's only per-key operator-only fence (ADR 0019 §3b), so the agent's own `set_config` cannot widen it. The fence is campaign's upload fence: + +- Symlinks are resolved **first**, and the real path must then be inside a root, so a link inside a root cannot point out of it. +- `..` is normalised away by that same resolution. +- Hardlinked files are refused, because a hardlink dodges every name check. +- Credential directory names (`.ssh`, `.aws`, …) and credential file names (`.env`, `*.pem`, `id_rsa`, …) are refused even inside a root. +- The agent's home (`~/.protoagent`, `$PROTOAGENT_HOME`) is refused. +- A root that is the filesystem root, the home directory or a parent of it, or the agent home is ignored as too broad. + +An empty `data_dirs` (the default) refuses every connect, and the refusal says where the operator sets it. Every source is **re-validated at query time**. A source whose file has left the fence, or whose root was removed from the setting, drops out of `allowed_paths` and is reported. A connect is not a permanent grant. + +### D5 — Charts are a new core artifact kind, `vega-lite` + +`show_artifact(kind="vega-lite", code=)` renders a Vega-Lite spec **with its data inline** (`data.values`). The data plugin's `data_chart` builds that spec: the model writes the encoding, and the tool inlines the query's rows. + +- **Vendoring.** `vega` 6.4.0, `vega-lite` 6.4.3 and `vega-embed` 7.3.0 (all BSD-3-Clause) are vendored as their own UMD builds, byte-for-byte. They are SRI-pinned in the shell's `LIB` map, served same-origin from the allowlisted `/plugins/artifact/vendor/` route with CORS for the opaque sandbox, and kept out of eol conversion by the existing `plugins/artifact/vendor/** -text`. The notices for everything the builds bundle are in `vendor/vega.LICENSES.txt`. +- **Sandbox.** The chart runs in the same no-same-origin frame as every artifact, under a **nonce CSP with no network**: `default-src 'none'`, nonce-only scripts, `connect-src 'none'`, `img-src`/`font-src` limited to `data:` (and `blob:` for images), and no workers, frames, objects, base or form targets. The injected base scripts (the ask bridge and the error boot) carry the nonce. +- **No `eval`.** Vega compiles expressions with `Function` by default. Here it runs as `vega-interpreter`, which vega-embed 7.3 bundles and selects with `ast: true`, so the CSP never needs `'unsafe-eval'`. +- **Loader lockdown.** Vega's loader is replaced with one that **refuses every load** (`load`/`sanitize`/`http`/`file`), so `data.url`, a spec loaded by URL, and image-mark URLs cannot fetch. Also, vega-embed merges a spec's own `usermeta.embedOptions` *over* the caller's options, loader included. The frame strips that key before embedding. `actions: false` removes the export/editor menu, whose editor action opens a remote site. +- **Feedback before render.** A spec that is not a JSON object, or that contains a `url` anywhere inside a data definition, is refused when the version is written: on create, update and rewrite, with a reason the model can act on. The frame re-checks all of it. The server-side check is feedback, and the frame is the fence. +- **Theme.** The chart is themed from the console's own data-viz tokens: `--pl-color-chart-series1…8` become the categorical range, `--pl-color-chart-axis` and `--pl-color-chart-grid` style the guides, and `--pl-color-fg`, `-fg-muted`, `-bg` and `--pl-font-sans` style the text and ground. A single view fills the panel width (`width: "container"`, `fit-x`). A live theme switch re-draws the chart in the new palette, with the tooltip theme following the ground's luminance. A spec's own `config` still overrides the theme. +- **Render verdict.** It is reported from the embed promise, not from the `load` event, because a chart can still fail after load. Dataflow errors reach the verdict through a logger. + +### D6 — Plugin services: a named-callable seam for plugin-to-plugin calls + +- **Provider side.** `registry.register_service(name, fn, description="")` offers `fn` as `.`. The name is namespaced by the provider's ID (the same rule as `emit`), lowercase, and refused when malformed. A duplicate keeps the first registration. +- **Consumer side.** `graph.sdk.service(name)` resolves the callable **at call time**, or returns `None` when no loaded plugin provides it. `None` is the normal answer for a disabled plugin, a plugin that is not installed, or a core that is too old. Consumers degrade with an actionable message. A service is an optional capability, never a hard dependency. +- **Wiring.** The loader aggregates services into `PluginLoadResult.services`. `server/plugin_wiring._apply_plugin_registries` rebinds the live table (`graph/plugin_services.py`) **wholesale** at build and on every reload, the same way verifiers and work providers are handled (#1752), so a disabled provider stops resolving immediately. The operator-MCP process runs that same function, so a tool running under ACP resolves services exactly as it would in the main process. +- **Testkit.** `testkit.FakeRegistry.register_service` captures services and raises on what the host would refuse. + +The first service is **`artifact.show(kind, code, title="") -> dict`**. It is `show_artifact` without code links: the same kinds, cap, checks, version chain and render-verdict wait. It returns `{ok, id, version, message, ref}`, and refusals are returned as data, never raised. `ref` is the `artifact-ref` chat-chip tail, which the caller appends last so the chat shows a chip that opens the chart. + +### D7 — Exports go only to the agent's workspace + +`data_export` writes to `/data-exports/` (`infra.paths.workspace_dir()`) and never to a source directory. The output is refused if the workspace itself sits inside a `data_dirs` root. + +- **Filename.** A bare name: path separators and `..` are refused. +- **How it writes.** A dedicated connection whose `allowed_paths` are the sources plus the *exact* output file runs `COPY () TO …`. + +## Consequences + +- **Faster charts.** A chart becomes a query plus a few hundred bytes of spec, instead of a component with the data typed into it. The send-to-chart time measured end to end is recorded in the core PR. +- **Engine-enforced read-only.** Both layers of the read-only guarantee are covered by plugin tests: write, `COPY`, `ATTACH`, `INSTALL`, `LOAD`, `SET`, `PRAGMA`, multi-statement batches, reads outside `data_dirs`, and symlink, `..` and hardlink escapes. +- **Vendored weight.** The artifact plugin gains about 830 KB of vendored JS. It loads only in a chart frame and is cached immutable. +- **Plugin services are a public API.** That raises the bar on providers: a service's signature and return shape are documented in its docstring and kept backward compatible. The plugin API reference pages document `register_service` and `sdk.service`. +- **Snapshots cost disk.** SQLite and Excel sources are snapshots, not live reads: a large SQLite database costs a Parquet copy in the plugin cache, and a write to the database is picked up at the next query, through mtime/size. +- **Version floor.** The data plugin's `min_protoagent_version` is the core release that ships this ADR's kind and seam. + +## Open questions (operator calls) + +- **Naming.** Is it "Data Analyst" or "Analyst"? The name matters most for the archetype and catalog row; the plugin can keep the ID `data` either way. +- **Archetype scope.** What does the Data Analyst bundle add on top of the plugin? Options include a SOUL, the two skills plus a "weekly metrics brief", a default `data_dirs` prompt as a bundle `config_input` on `data.data_dirs`, and a dashboard view that pins several charts. +- **Dashboards.** Several pinned charts with shared filters would mean either a `vega` (full grammar) kind or a dashboard artifact. That decision is deferred until there is a real use case. diff --git a/docs/adr/index.md b/docs/adr/index.md index 9eb6067c3..4eea4bc29 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -123,3 +123,4 @@ decision, numbered, never deleted (supersede instead). | [0112](./0112-console-code-pane.md) | Console code pane (**amended 2026-09-25: an opt-in toolset, `filesystem.code_pane`, off by default**) — a read-only **navigator** surface the agent points into, motivated by evidence that watching an agent work lowers comprehension (arXiv 2607.26375; Anthropic's AI-assistance coding-skills study). (D1) core `code` surface, File \| Diff tabs + a Recent trail, opened on the dock NOT holding chat; (D2) `GET /api/fs/file` — the `read_file` fence, `\n`-delimited lines with endings preserved, 2 MB / 20k lines / 2k chars-per-line caps with paging, binary → metadata; (D3) `GET /api/fs/diff` — hardened git (no ext diff / textconv / filter driver / fsmonitor / hook / submodule recursion; all `GIT_*` env scrubbed), scoped to the project root, untracked files as Python-built patches, 1 MB cap, 10 s → 504; (D4) `tools/fs_secrets.py` secret-like-name deny list for display surfaces (403 before existence; `read_file` unchanged — a follow-up); (D5) `show_code` fs tool → a strictly validated `code-ref` component; (D6) console: chip, live-only auto-open on desktop, never on mobile, opt-in debounced follow mode with pin, pane-by-default links; highlighting library decided in the console PR. Amends 0035/0051/0056/0062/0086. | Proposed | | [0113](./0113-agent-pairing-for-remote-fleet-members.md) | **Agent pairing for remote fleet members.** A hub pairs with a remote the way a phone does (ADR 0087): the remote shows a code (Settings ▸ Devices ▸ Pair an agent, or `protoagent pair`), and the hub claims it through the existing unauthenticated `/api/pairing/claim`, storing the minted per-hub device token (revocable on the remote) as the remote's bearer. (D2) agent codes are 10-char Crockford base32 (~50 bits), 5-min TTL, single-use, under the shared 5-miss lockout, which resets on each new code; the kind is fixed when the code is minted. (D4) a delegate to a remote routes through the hub's loopback proxy (fleet token → hub → paired token), so there is one token and one registry. (D5) an authenticated probe reports `auth: ok\|rejected`. (D6) remote WebSockets are re-enabled, and the hub never lends a stored credential: it only swaps a hub-verified operator `?token=`, and passes tickets through. (D8) the `settings.devices` flag is removed after a desktop-app test. (D9) mDNS advertises only on a reachable bind. (D10) a credential goes over plain http only to loopback or a tailnet address, or with an explicit opt-in. Rejects hub-requests/remote-approves and HTTPS discovery. Amends 0042 §I and 0087. | Proposed | | [0115](./0115-gateway-inflight-limiter.md) | **Model calls share a bounded, priority-ordered in-flight lane per endpoint and model (off by default).** Nothing caps how many chat-model calls are in flight at once (`graph/` has no semaphore; the per-fan-out caps don't know about each other), so under load every caller times out and retries together (the #209 review-gate incident). A new in-process limiter in `graph/llm_limiter.py` keyed by `(base_url, model)` gives each lane a bounded queue: `model.max_inflight` (default **0 = off**), `model.inflight_queue_timeout` (300 s, separate from `request_timeout`), `model.inflight_interactive_reserve` (slots only `interactive` callers take). Waiters are served strict-priority (interactive > default > bulk) with 60 s aging, acquire OUTSIDE the stream-stall guard so wait time never counts as backend latency, and a queue timeout raises `GatewayQueueTimeout` (not retried, never mislabelled a stall). Per-process only (ACP coders / clawpatch / other instances stay out, D7); gateway-side limits remain the cross-process backstop. Adds queue-depth/wait Prometheus metrics + `GET /api/telemetry/llm-lanes` + `sdk.llm_lanes()`. Builds on 0006, keyed per 0106, boundary per 0025. Refs #3760. | Proposed | +| [0116](./0116-local-first-data-analyst-duckdb-and-vega-lite-charts.md) | **A local-first data analyst: DuckDB over files in place, charts as Vega-Lite specs, and a plugin-service call seam.** (D1) a `data` plugin in its own repo (`protoLabsAI/data-plugin`) — `data_connect/sources/schema/query/profile/chart/export` + two skills; the archetype bundle comes later. (D2) embedded DuckDB views over CSV/Parquet/JSON in place; SQLite/XLSX snapshotted to Parquet (their extensions would `INSTALL` from the network). (D3) read-only enforced BY THE ENGINE — a fresh in-memory connection per query with exact-file `allowed_paths`, `enable_external_access=false`, a private temp dir, extension autoload/autoinstall off, `lock_configuration` — plus a single-`SELECT` statement guard and row/time/memory caps. (D4) the scope is the operator-only `data_dirs` (`spawns: true`), campaign's upload fence (symlinks resolved first, credential names, agent home, hardlinks, too-broad roots), re-validated per query. (D5) a core `vega-lite` artifact kind: vendored SRI-pinned vega/vega-lite/vega-embed, a nonce CSP with no network, the CSP-safe expression interpreter (no eval), a deny-everything loader (+ `usermeta.embedOptions` stripped), inline-data checks at write time, themed from `--pl-color-chart-*` with live re-theme. (D6) plugin services — `registry.register_service` / `sdk.service(name)`, rebound wholesale on reload; first service `artifact.show`. (D7) exports only into the agent workspace. | Proposed | diff --git a/docs/reference/plugin-registry-api.md b/docs/reference/plugin-registry-api.md index 91a21a6ff..116e81107 100644 --- a/docs/reference/plugin-registry-api.md +++ b/docs/reference/plugin-registry-api.md @@ -68,6 +68,7 @@ Host services (agent invoke + event bus) a surface/route can use — the server | [`register_mcp_server()`](#registry-register-mcp-server) | Contribute a **managed MCP server** the agent connects to ([ADR 0019](/adr/0019-plugin-config-settings-secrets)) | | [`register_middleware()`](#registry-register-middleware) | Add a plugin-contributed LangGraph `AgentMiddleware` ([ADR 0032](/adr/0032-pluggable-middleware)) | | [`register_router()`](#registry-register-router) | Mount a FastAPI `APIRouter` on the server ([ADR 0018](/adr/0018-plugin-surfaces-routes-subagents)) | +| [`register_service()`](#registry-register-service) | Offer `fn` to OTHER plugins as the service `.` ([ADR 0116](/adr/0116-local-first-data-analyst-duckdb-and-vega-lite-charts)) | | [`register_setup_step()`](#registry-register-setup-step) | Register a SETUP STEP: the server-side half of a `plugin_setup` setup-gap action | | [`register_skill_dir()`](#registry-register-skill-dir) | Add a directory of `SKILL.md` skills bundled with the plugin | | [`register_subagent()`](#registry-register-subagent) | Add a `SubagentConfig` to `SUBAGENT_REGISTRY` ([ADR 0018](/adr/0018-plugin-surfaces-routes-subagents)) | @@ -351,6 +352,31 @@ plugin-views.md, [ADR 0026](/adr/0026-plugin-contributed-console-surfaces)); a p [#1732](https://github.com/protoLabsAI/protoAgent/issues/1732)). The default-deny auth middleware guards all non-public paths regardless of prefix. +### `registry.register_service` {#registry-register-service} + +```python +registry.register_service(name: str, fn, description: str = '') -> None +``` + +Offer `fn` to OTHER plugins as the service `.` ([ADR 0116](/adr/0116-local-first-data-analyst-duckdb-and-vega-lite-charts)). + +The cross-plugin CALL seam: where the event bus (`emit`) is fire-and-forget, a +service returns a result. A consumer resolves it at call time with +`graph.sdk.service(".")` and gets your callable — or `None` when +your plugin is disabled, so it must degrade (a service is an optional capability). It +never imports your plugin, so you can refactor freely behind the name. The artifact +plugin's `artifact.show` is the reference: the data plugin creates charts through it. + +The name is namespaced to this plugin: `"show"` registers `".show"`, and +a name already under this plugin's namespace is kept as-is (a plugin may only provide +under its own). The bare part is lowercase `[a-z][a-z0-9_]*`. Treat the callable's +signature and return shape as a PUBLIC API — document them in its docstring, keep them +backward compatible, and return refusals as data rather than raising for an expected +"no". It runs in the CALLER's thread (often a tool body; sync or async is your call, +say which). `description` is a one-line summary for status surfaces. Live while the +plugin is loaded — a reload that disables it stops the name resolving. Guard with +`getattr(registry, "register_service", None)` on hosts older than this seam. + ### `registry.register_setup_step` {#registry-register-setup-step} ```python diff --git a/docs/reference/plugin-sdk-api.md b/docs/reference/plugin-sdk-api.md index 3fdb14170..58ab825cd 100644 --- a/docs/reference/plugin-sdk-api.md +++ b/docs/reference/plugin-sdk-api.md @@ -23,6 +23,8 @@ the module's own. ## Contents +**Plugin services (the plugin↔plugin call seam, ADR 0116)** — [`service()`](#sdk-service) + **Agent + model access (the plugin↔agent channel, ADR 0043)** — [`complete()`](#sdk-complete), [`config()`](#sdk-config), [`gateway_client()`](#sdk-gateway-client), [`run_subagent()`](#sdk-run-subagent), [`subagent_types()`](#sdk-subagent-types), [`turn_stop_reason()`](#sdk-turn-stop-reason) **Model in-flight priority (the plugin↔limiter channel, ADR 0115 D6)** — [`llm_lanes()`](#sdk-llm-lanes), [`llm_priority()`](#sdk-llm-priority) @@ -47,6 +49,32 @@ the module's own. **Managed Python runtime (ADR 0094 — the provisioned child interpreter)** — [`managed_python_exe()`](#sdk-managed-python-exe) +## Plugin services (the plugin↔plugin call seam, ADR 0116) + +### `sdk.service` {#sdk-service} + +```python +sdk.service(name: str) -> Callable | None +``` + +The callable another plugin offers as `name` (`"."`), or `None`. + +The way one plugin CALLS another without importing it: the provider registers it with +`registry.register_service` and you resolve it here, at call time — never at +`register()` time, since plugins load in no guaranteed order and a reload can swap the +provider in or out. `None` means no loaded plugin provides it (disabled, not installed, +or a core older than the provider needs): degrade with an actionable message rather than +fail. Guard the lookup itself with `getattr(sdk, "service", None)` on cores older than +this seam. The callable's signature and return shape are the PROVIDER's documented API — +read its docstring (`help(sdk.service("artifact.show"))`). The reference provider is the +artifact plugin's `artifact.show`:: + + show = sdk.service("artifact.show") + if show is None: + return "The Artifact plugin is off — enable it to see charts." + r = show(kind="vega-lite", code=spec_json, title="Sales by weekday") + return r["message"] + r["ref"] # the artifact-ref chip tail goes LAST + ## Agent + model access (the plugin↔agent channel, ADR 0043) ### `sdk.config` {#sdk-config} diff --git a/graph/plugin_services.py b/graph/plugin_services.py new file mode 100644 index 000000000..9ec08ecf8 --- /dev/null +++ b/graph/plugin_services.py @@ -0,0 +1,70 @@ +"""Plugin services (ADR 0116) — a callable one plugin offers and another calls BY NAME. + +The cross-plugin call seam. Before it, a plugin could reach another only through the event bus +(ADR 0039), which is fire-and-forget: no return value, and no answer to "is that plugin even +on?". A plugin that needs a RESULT from another — the data plugin creating a chart in the +Artifact panel and reporting the artifact it made — had nothing to call except the other +plugin's internals, which is exactly the coupling the plugin contract forbids. + +So a plugin registers a service (``registry.register_service("show", fn)`` → ``artifact.show``) +and a consumer resolves it at CALL time (``graph.sdk.service("artifact.show")``), getting the +callable or ``None``. ``None`` is the normal "that plugin is disabled / not installed / on an +older core" answer, and every consumer must handle it — a service is an optional capability, +never a hard dependency (a hard one is a manifest ``requires_plugins`` concern, not this). + +The live mapping is replaced WHOLESALE at build and on every plugin reload, like the verifier +and work-provider registries (#1752: a stale registry is worse than an empty one — a disabled +plugin's service must stop resolving the moment it's unloaded, not at the next restart). The +operator-MCP process applies the same mapping (``server/operator_mcp.py``), so a plugin tool +running there resolves services exactly as it would in the main process. +""" + +from __future__ import annotations + +import logging +import re +from collections.abc import Callable, Mapping + +log = logging.getLogger(__name__) + +# ``.`` — the provider's id is the namespace (``register_service`` prefixes it), +# so one plugin can't register a name under another's. Same id alphabet as the manifest. +SERVICE_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*\.[a-z][a-z0-9_]*$") + +# name -> callable. Rebound wholesale (one GIL-atomic assignment) by ``set_plugin_services``. +_SERVICES: dict[str, Callable] = {} +# name -> {"plugin_id", "description"}; parallel to _SERVICES so its values stay THE callable. +_SERVICE_META: dict[str, dict] = {} + + +def is_service_name(name: object) -> bool: + """True for a well-formed ``.`` service name.""" + return isinstance(name, str) and len(name) <= 128 and bool(SERVICE_NAME_RE.match(name)) + + +def set_plugin_services(mapping: Mapping[str, Callable] | None, meta: Mapping[str, dict] | None = None) -> None: + """Replace the live service set (called at build + on every plugin reload). + + Wholesale replacement, not a merge: a reload that drops a plugin drops its services too. + Malformed names and non-callables are skipped (the registry already refuses them; this is + the second line for a duck-typed bundle).""" + global _SERVICES, _SERVICE_META + fresh = {n: fn for n, fn in (mapping or {}).items() if is_service_name(n) and callable(fn)} + _SERVICES = fresh + _SERVICE_META = {n: dict(m) for n, m in (meta or {}).items() if n in fresh} + + +def get_service(name: str) -> Callable | None: + """The callable registered as ``name``, or ``None`` when no loaded plugin provides it.""" + return _SERVICES.get(name) if isinstance(name, str) else None + + +def service_names() -> list[str]: + """Registered service names, sorted — the introspection half, for a status surface.""" + return sorted(_SERVICES) + + +def service_meta(name: str) -> dict | None: + """``{"plugin_id", "description"}`` for a registered service, or ``None``.""" + m = _SERVICE_META.get(name) + return dict(m) if m is not None else None diff --git a/graph/plugins/loader.py b/graph/plugins/loader.py index ee56f22e0..bf6d35e12 100644 --- a/graph/plugins/loader.py +++ b/graph/plugins/loader.py @@ -64,6 +64,8 @@ class PluginLoadResult: thread_id_resolver: object = None # (request_metadata, session_id) -> str (#571); last plugin wins chat_commands: dict = field(default_factory=dict) # token -> handler; user-only chat control commands components: dict = field(default_factory=dict) # component-v1 kind -> props validator (#3617) + services: dict = field(default_factory=dict) # "." -> callable (ADR 0116) + service_meta: dict = field(default_factory=dict) # name -> {plugin_id, description} meta: list[dict] = field(default_factory=list) @@ -1096,6 +1098,25 @@ def load_plugins(config, *, core_tool_names: set[str] | None = None) -> PluginLo log.warning("[plugins] %s: component %s collides — skipped", manifest.id, name) continue result.components[name] = validator + for name, fn in getattr(registry, "services", {}).items(): # plugin services (ADR 0116) + # register_service namespaces every name, but `registry.services` is a plain dict a + # plugin could write to directly — so the loader re-checks: a plugin provides ONLY + # under its own id, never as (or over) another plugin's service. + if not name.startswith(f"{manifest.id}."): + log.warning( + "[plugins] %s: service %s is outside this plugin's namespace (%s.*) — skipped", + manifest.id, + name, + manifest.id, + ) + continue + if name in result.services: # namespaced per plugin, so only a duplicate plugin id hits this + log.warning("[plugins] %s: service %s collides — skipped", manifest.id, name) + continue + result.services[name] = fn + smeta = getattr(registry, "service_meta", {}).get(name) + if smeta: + result.service_meta[name] = smeta entry["loaded"] = True entry["tools"] = [t.name for t in kept] # Count the conventional skills/ dir too (auto-discovered above) — counting only @@ -1108,6 +1129,7 @@ def load_plugins(config, *, core_tool_names: set[str] | None = None) -> PluginLo entry["mcp_servers"] = len(registry.mcp_servers) entry["chat_commands"] = [f"/{t}" for t in registry.chat_commands] entry["components"] = sorted(getattr(registry, "components", {})) + entry["services"] = sorted(getattr(registry, "services", {})) result.meta.append(entry) log.info( "[plugins] loaded %s: %d tool(s), %d skill dir(s), %d route(s), " diff --git a/graph/plugins/registry.py b/graph/plugins/registry.py index e42c627cc..a46e6ae68 100644 --- a/graph/plugins/registry.py +++ b/graph/plugins/registry.py @@ -42,6 +42,8 @@ class PluginRegistry: - ``components`` — component-v1 kinds + their props validators, emitted by the plugin's own tools and rendered by a console chat-component renderer (``register_component``, #3617). + - ``services`` — named callables OTHER plugins call through ``graph.sdk.service`` + (``register_service``, ADR 0116). Routes and surfaces both wire at process init. Routes now ALSO hot-mount on a config reload — a newly-enabled plugin's routers, public paths, verifiers, and @@ -108,6 +110,8 @@ def __init__( self.embedders: dict = {} # name -> (config) -> (text -> vector) embed_fn (ADR 0031) self.chat_commands: dict = {} # token -> async (rest, session_id) -> str|None (user-only control commands) self.components: dict = {} # component-v1 kind -> props validator (#3617) + self.services: dict = {} # "." -> callable, other plugins' sdk.service (ADR 0116) + self.service_meta: dict = {} # name -> {"plugin_id", "description"}; parallel, see goal_verifier_meta def report_setup_gap(self, key: str, message: str | None, *, label: str | None = None, action=None) -> None: """Tell the operator this plugin can't do its job until something is fixed @@ -297,6 +301,43 @@ def register_component(self, name: str, validator) -> None: return self.components[name] = validator + def register_service(self, name: str, fn, description: str = "") -> None: + """Offer ``fn`` to OTHER plugins as the service ``.`` (ADR 0116). + + The cross-plugin CALL seam: where the event bus (:meth:`emit`) is fire-and-forget, a + service returns a result. A consumer resolves it at call time with + ``graph.sdk.service(".")`` and gets your callable — or ``None`` when + your plugin is disabled, so it must degrade (a service is an optional capability). It + never imports your plugin, so you can refactor freely behind the name. The artifact + plugin's ``artifact.show`` is the reference: the data plugin creates charts through it. + + The name is namespaced to this plugin: ``"show"`` registers ``".show"``, and + a name already under this plugin's namespace is kept as-is (a plugin may only provide + under its own). The bare part is lowercase ``[a-z][a-z0-9_]*``. Treat the callable's + signature and return shape as a PUBLIC API — document them in its docstring, keep them + backward compatible, and return refusals as data rather than raising for an expected + "no". It runs in the CALLER's thread (often a tool body; sync or async is your call, + say which). ``description`` is a one-line summary for status surfaces. Live while the + plugin is loaded — a reload that disables it stops the name resolving. Guard with + ``getattr(registry, "register_service", None)`` on hosts older than this seam.""" + from graph.plugin_services import is_service_name + + pid = self.plugin_id + key = name if isinstance(name, str) and name.startswith(f"{pid}.") else f"{pid}.{name}" + if not is_service_name(key) or not callable(fn): + log.warning( + "[plugins] %s: service %r refused — the name must be lowercase [a-z][a-z0-9_]* " + "(namespaced to this plugin) and fn callable", + pid, + name, + ) + return + if key in self.services: + log.warning("[plugins] %s: service %s registered twice — keeping the first", pid, key) + return + self.services[key] = fn + self.service_meta[key] = {"plugin_id": pid, "description": (description or "").strip()} + def emit(self, topic: str, data: dict | None = None) -> None: """Broadcast an event on the bus (ADR 0039) — fire-and-forget. diff --git a/graph/plugins/testkit.py b/graph/plugins/testkit.py index 698f811c0..71b7ad379 100644 --- a/graph/plugins/testkit.py +++ b/graph/plugins/testkit.py @@ -281,6 +281,8 @@ def __init__( self.embedders: dict = {} self.chat_commands: dict = {} # slugified token -> handler self.components: dict = {} # component-v1 kind -> props validator (#3617) + self.services: dict = {} # "." -> callable (ADR 0116) + self.service_meta: dict = {} # name -> {plugin_id, description} self.late_tool_factories: list = [] self.saved_media: list = [] # (data, mime, meta) — save_media captures (#1929) self.handlers: dict = {} # topic -> [handlers] @@ -333,6 +335,23 @@ def register_component(self, name: str, validator) -> None: the host also refuses a core/invalid name.""" self.components[name] = validator + def register_service(self, name: str, fn, description: str = "") -> None: + """Capture a plugin service under its namespaced name (``self.services["."]``), + with the host's validation — a malformed name or non-callable raises here instead of + being dropped with a warning, so a registration the host would refuse fails the test. + To exercise a CONSUMER, put fakes in the live table instead: + ``graph.plugin_services.set_plugin_services({"artifact.show": fake})``.""" + from graph.plugin_services import is_service_name + + pid = self.plugin_id + key = name if isinstance(name, str) and name.startswith(f"{pid}.") else f"{pid}.{name}" + if not is_service_name(key) or not callable(fn): + raise ValueError(f"service {name!r} would be refused by the host (bad name or non-callable)") + if key in self.services: + raise ValueError(f"service {key} registered twice — the host keeps only the first") + self.services[key] = fn + self.service_meta[key] = {"plugin_id": pid, "description": description} + def register_chat_command(self, name: str, handler) -> None: """Capture a user-only ``/`` control command — with the real registry's slugify + validation, so a registration the host would refuse (empty/unslugifiable diff --git a/graph/sdk.py b/graph/sdk.py index 09312754c..33def994d 100644 --- a/graph/sdk.py +++ b/graph/sdk.py @@ -62,6 +62,33 @@ from graph.components import encode_component # noqa: F401 +# ── plugin services (the plugin↔plugin call seam, ADR 0116) ───────────────────────────── + + +def service(name: str) -> Callable | None: + """The callable another plugin offers as ``name`` (``"."``), or ``None``. + + The way one plugin CALLS another without importing it: the provider registers it with + ``registry.register_service`` and you resolve it here, at call time — never at + ``register()`` time, since plugins load in no guaranteed order and a reload can swap the + provider in or out. ``None`` means no loaded plugin provides it (disabled, not installed, + or a core older than the provider needs): degrade with an actionable message rather than + fail. Guard the lookup itself with ``getattr(sdk, "service", None)`` on cores older than + this seam. The callable's signature and return shape are the PROVIDER's documented API — + read its docstring (``help(sdk.service("artifact.show"))``). The reference provider is the + artifact plugin's ``artifact.show``:: + + show = sdk.service("artifact.show") + if show is None: + return "The Artifact plugin is off — enable it to see charts." + r = show(kind="vega-lite", code=spec_json, title="Sales by weekday") + return r["message"] + r["ref"] # the artifact-ref chip tail goes LAST + """ + from graph.plugin_services import get_service + + return get_service(name) + + # ── agent + model access (the plugin↔agent channel, ADR 0043) ────────────────────────── diff --git a/plugins/artifact/README.md b/plugins/artifact/README.md index 0b707fe28..43bcccf34 100644 --- a/plugins/artifact/README.md +++ b/plugins/artifact/README.md @@ -27,9 +27,10 @@ and mounts its console view live, with no restart. - **Tools** — an artifact is a **version chain** (the Claude "update vs rewrite" model), so editing iterates the same artifact instead of flooding the panel with near-duplicates: - `show_artifact(kind, code, title, links?)` — **create** (`kind` ∈ `html` · `markdown` · `svg` · - `mermaid` · `react`). `markdown` renders with design-system prose styling (` ```mermaid ` fences - become live diagrams); `react` can `import` the curated libraries below. `links` (mermaid) — - see **Code-linked diagrams** below. + `mermaid` · `vega-lite` · `react`). `markdown` renders with design-system prose styling + (` ```mermaid ` fences become live diagrams); `vega-lite` is a **chart** — see **Charts** below; + `react` can `import` the curated libraries below. `links` (mermaid) — see **Code-linked + diagrams** below. - `update_artifact(old_string, new_string, artifact_id?, links?)` — **targeted edit** (string-replace, must match once) → new version. The fast path for small changes. A diagram's links carry over unless `links` is passed (`{}` clears them). @@ -154,6 +155,36 @@ are injected into every `html` / `react` / `markdown` artifact (via the host-ser `/_ds/plugin-kit.css`), so even plain elements (`className="pl-btn pl-btn--primary"`) follow the live theme — reach for a raw element with `className="pl-…"` for any component `@pl/ui` doesn't wrap. +## Charts (`vega-lite`, ADR 0116) + +A chart is a [Vega-Lite](https://vega.github.io/vega-lite/) spec with its rows **inline** in +`data.values` — a few hundred bytes of JSON instead of a hand-written component, which is what +makes a chart fast to produce. The panel draws it with the vendored `vega` 6.4.0 / `vega-lite` +6.4.3 / `vega-embed` 7.3.0 (BSD-3-Clause; their own UMD builds, byte-for-byte, notices in +`vendor/vega.LICENSES.txt`), SRI-pinned like the other UMD libs. + +- **Themed for you.** The chart takes the console's own data-viz tokens — `--pl-color-chart-series1…8` + as the categorical palette, `--pl-color-chart-axis`/`-grid` for guides, the theme's fg/bg/font — + and redraws on a live theme switch. Leave colours and background out of the spec; a spec's own + `config` still wins when you mean one. A single view fills the panel width. +- **Inline data only.** A `url` inside any data definition is refused when the version is written + (create, update and rewrite), with a reason. In the frame, Vega's loader refuses every load + (`data.url`, a spec by URL, image marks), the CSP has no network (`connect-src 'none'`, images and + fonts `data:` only), expressions run as the CSP-safe interpreter (no `eval`, so no + `'unsafe-eval'`), and a spec's `usermeta.embedOptions` — which vega-embed would otherwise let + override the embed options, loader included — is stripped. No export/editor action menu. +- **From another plugin.** The `artifact.show` plugin service creates one without importing this + plugin — `graph.sdk.service("artifact.show")(kind="vega-lite", code=spec_json, title=…)` → `{ok, + id, version, message, ref}`; append `ref` (the chat chip) last. The data plugin's `data_chart` + runs a query and hands its rows here this way. + +```json +{"title": "Revenue by weekday", "mark": {"type": "bar", "tooltip": true}, + "encoding": {"x": {"field": "weekday", "type": "nominal", "sort": "-y"}, + "y": {"field": "revenue", "type": "quantitative", "axis": {"format": "$,.0f"}}}, + "data": {"values": [{"weekday": "Sat", "revenue": 2310.5}, {"weekday": "Fri", "revenue": 1876}]}} +``` + ## Slide decks (.pptx) A `.pptx` saved with `save_file_artifact` previews as its **real slides**, not a text dump: a large @@ -264,7 +295,7 @@ the page announces it is listening (`protoagent:ready`), targeted at the page's > **Offline / no network.** Everything is **vendored** under `vendor/` and served same-origin from > `/plugins/artifact/vendor/…`, so every artifact kind renders **fully offline** — no `cdnjs`, no > outbound network at all (`capabilities.network: []` is literally true): -> - **UMD `