Skip to content

Commit fd6791c

Browse files
hyperpolymathclaude
andcommitted
feat(pool): persistent Deno worker pool eliminates JS cold-start overhead
Adds BojRest.JsWorkerPool (Supervisor) and BojRest.JsWorker (GenServer) for persistent Deno process management via stdin/stdout newline-delimited JSON. priv/js_pool_worker.js caches imported mod.js handles across calls. Worker selection uses consistent hashing on mod_js_path for module-cache locality. Pool size defaults to 5 (js_worker_pool_size config). Falls back to JsInvoker fork-per-call when Deno or the pool script is absent. Router now routes JS invocations through JsWorkerPool.invoke/4 instead of JsInvoker.invoke/4 directly. Credentials are forwarded inside the JSON request and applied/cleaned per-invocation on the Deno side (safe because each worker processes one request at a time). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 4e50e01 commit fd6791c

6 files changed

Lines changed: 336 additions & 7 deletions

File tree

ROADMAP.adoc

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -82,15 +82,16 @@ metadata parity before Grade C dogfooding.
8282

8383
JS cartridge dispatch (the large remaining gap — 99 of 106 cartridges use `mod.js`, not Zig `.so`):
8484

85-
* [ ] `BojRest.JsInvoker` — shells out to `deno run mod.js` per invocation (Phase 1: fork-per-call, ~50 ms overhead acceptable for initial traffic)
86-
* [ ] Router dispatch branch: if `cart["ffi"]` key present → `Invoker` (Zig path); else → `JsInvoker` (Deno path)
87-
* [ ] Fix `cartridge_so_path/1` to read `cart["ffi"]["so_path"]` from the manifest instead of hardcoding the derivation
85+
* [x] `BojRest.JsInvoker` — shells out to `deno run mod.js` per invocation (Phase 1: fork-per-call, ~200 ms cold-start)
86+
* [x] Router dispatch branch: if `cart["ffi"]` key present → `Invoker` (Zig path); else → `JsWorkerPool` (Deno pool path)
87+
* [x] Fix `cartridge_so_path/1` to read `cart["ffi"]["so_path"]` from the manifest instead of hardcoding the derivation
88+
* [x] `BojRest.JsWorkerPool` + `BojRest.JsWorker` — persistent Deno worker pool (Phase 2); consistent hash on mod path for module-cache locality; `JsInvoker` fork-per-call as fallback when pool absent
8889
* [ ] End-to-end test: `curl POST /cartridge/boj-health/invoke` (Zig) + `curl POST /cartridge/local-memory-mcp/invoke` (JS)
8990

9091
Gateway sidecar (`http-capability-gateway` in front of boj-server):
9192

92-
* [ ] Add `gateway-policy.yaml` (done — `container/gateway-policy.yaml`) to dev compose as optional sidecar
93-
* [ ] Implement `PolicyLoader.load_from_boj_catalog/1` in `http-capability-gateway` — reads `BOJ_CARTRIDGES_ROOT/*/cartridge.json` at boot, infers per-cartridge invoke exposure from `auth.method` (`none` → `public`, `api-key`/`bearer` → `authenticated`). Zero manual policy maintenance thereafter. See `docs/gateway-catalog-integration.adoc`.
93+
* [x] Add `gateway-policy.yaml` (done — `container/gateway-policy.yaml`) to dev compose as optional sidecar
94+
* [x] Implement `PolicyLoader.load_from_boj_catalog/1` in `http-capability-gateway` — reads `BOJ_CARTRIDGES_ROOT/*/cartridge.json` at boot, infers per-cartridge invoke exposure from `auth.method` (`none` → `public`, others → `authenticated`). Zero manual policy maintenance thereafter. See `docs/gateway-catalog-integration.adoc`.
9495
* [ ] Update `compose.dev.yaml` with gateway sidecar entry (port 7800 → boj-rest:7700)
9596
* [ ] Retire static `gateway-policy.yaml` once catalog mode is live
9697

elixir/lib/boj_rest/application.ex

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ defmodule BojRest.Application do
2222
children =
2323
[
2424
{BojRest.NodeKey, data_dir: data_dir},
25-
{BojRest.Catalog, cartridges_root: cartridges_root}
25+
{BojRest.Catalog, cartridges_root: cartridges_root},
26+
{BojRest.JsWorkerPool, []}
2627
] ++
2728
if start_server do
2829
[{Plug.Cowboy, scheme: :http, plug: BojRest.Router, options: [port: port]}]

elixir/lib/boj_rest/js_worker.ex

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# SPDX-License-Identifier: PMPL-1.0-or-later
2+
# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
3+
defmodule BojRest.JsWorker do
4+
@moduledoc """
5+
GenServer wrapping one persistent Deno pool-worker process.
6+
7+
Communication: newline-delimited JSON over the process's stdin/stdout.
8+
Requests are processed sequentially on the Deno side (one `await` at a time),
9+
but multiple callers can queue requests concurrently — replies are matched by
10+
the `id` field in each response.
11+
12+
When the underlying Deno process exits unexpectedly all pending callers
13+
receive `{:error, %{classification: :worker_crashed, ...}}` and the GenServer
14+
stops so the pool Supervisor can restart it with a fresh Deno process.
15+
"""
16+
use GenServer
17+
require Logger
18+
19+
# Budget slightly above the JsInvoker timeout so the caller sees a clean
20+
# timeout error rather than a GenServer call timeout exception.
21+
@timeout_ms 30_000
22+
@call_timeout_ms @timeout_ms + 5_000
23+
24+
defstruct [:port, :pending, :buffer, :id_counter]
25+
26+
# ── public API ───────────────────────────────────────────────────────────────
27+
28+
def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: Keyword.get(opts, :name))
29+
30+
@doc """
31+
Invoke `tool_name` on the cartridge at `mod_js_path` via the persistent worker.
32+
33+
`extra_env` is forwarded inside the JSON request and applied/cleared by the
34+
pool worker per invocation — safe because the worker processes requests one
35+
at a time.
36+
"""
37+
@spec invoke(GenServer.server(), String.t(), String.t(), map(), map()) ::
38+
{:ok, map()} | {:error, map()}
39+
def invoke(server, mod_js_path, tool_name, args, extra_env \\ %{}) do
40+
GenServer.call(server, {:invoke, mod_js_path, tool_name, args, extra_env}, @call_timeout_ms)
41+
end
42+
43+
# ── GenServer callbacks ────────────────────────────────────────────────────
44+
45+
@impl true
46+
def init(opts) do
47+
deno = Keyword.fetch!(opts, :deno_path)
48+
runner = Keyword.fetch!(opts, :runner_path)
49+
50+
port =
51+
Port.open({:spawn_executable, deno}, [
52+
:binary,
53+
:exit_status,
54+
{:args, ["run", "--allow-net", "--allow-env", "--allow-read", runner]}
55+
])
56+
57+
Logger.debug("JsWorker started", deno: deno, runner: runner)
58+
{:ok, %__MODULE__{port: port, pending: %{}, buffer: "", id_counter: 0}}
59+
end
60+
61+
@impl true
62+
def handle_call({:invoke, mod_js_path, tool_name, args, extra_env}, from, state) do
63+
id = Integer.to_string(state.id_counter)
64+
65+
request =
66+
Jason.encode!(%{
67+
id: id,
68+
mod: mod_js_path,
69+
tool: tool_name,
70+
args: args,
71+
env: extra_env
72+
})
73+
74+
Port.command(state.port, request <> "\n")
75+
timer = Process.send_after(self(), {:timeout, id}, @timeout_ms)
76+
pending = Map.put(state.pending, id, {from, timer})
77+
{:noreply, %{state | pending: pending, id_counter: state.id_counter + 1}}
78+
end
79+
80+
@impl true
81+
def handle_info({port, {:data, data}}, %{port: port} = state) do
82+
full = state.buffer <> data
83+
{lines, rest} = split_lines(full)
84+
new_state = Enum.reduce(lines, state, &dispatch_response/2)
85+
{:noreply, %{new_state | buffer: rest}}
86+
end
87+
88+
def handle_info({port, {:exit_status, code}}, %{port: port} = state) do
89+
Logger.error("Deno pool worker exited unexpectedly", exit_code: code)
90+
91+
Enum.each(state.pending, fn {_id, {from, timer}} ->
92+
Process.cancel_timer(timer)
93+
GenServer.reply(from, {:error, %{classification: :worker_crashed, body: "Deno worker exited with code #{code}"}})
94+
end)
95+
96+
{:stop, :normal, %{state | pending: %{}}}
97+
end
98+
99+
def handle_info({:timeout, id}, state) do
100+
case Map.pop(state.pending, id) do
101+
{{from, _timer}, pending} ->
102+
GenServer.reply(from, {:error, %{classification: :timeout, body: "JS invocation timed out after #{@timeout_ms}ms"}})
103+
{:noreply, %{state | pending: pending}}
104+
105+
{nil, _} ->
106+
{:noreply, state}
107+
end
108+
end
109+
110+
# ── private helpers ────────────────────────────────────────────────────────
111+
112+
# Split accumulated bytes at newlines; return {complete_lines, remainder}.
113+
defp split_lines(data) do
114+
parts = String.split(data, "\n")
115+
{Enum.slice(parts, 0, length(parts) - 1) |> Enum.reject(&(&1 == "")),
116+
List.last(parts) || ""}
117+
end
118+
119+
defp dispatch_response(line, state) do
120+
case Jason.decode(line) do
121+
{:ok, %{"id" => id, "status" => status, "data" => data}} ->
122+
case Map.pop(state.pending, id) do
123+
{{from, timer}, pending} ->
124+
Process.cancel_timer(timer)
125+
result = if status in 200..299, do: {:ok, data}, else: {:error, %{classification: :js_error, body: data, status: status}}
126+
GenServer.reply(from, result)
127+
%{state | pending: pending}
128+
129+
{nil, _} ->
130+
Logger.warning("JsWorker: unexpected response id", id: id)
131+
state
132+
end
133+
134+
_ ->
135+
Logger.warning("JsWorker: could not decode response line", line: line)
136+
state
137+
end
138+
end
139+
end
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# SPDX-License-Identifier: PMPL-1.0-or-later
2+
# Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
3+
defmodule BojRest.JsWorkerPool do
4+
@moduledoc """
5+
Supervisor managing a fixed pool of persistent Deno worker processes.
6+
7+
Each worker (`BojRest.JsWorker`) holds one live Deno process and handles
8+
invocations via stdin/stdout JSON. Worker selection uses consistent hashing
9+
on `mod_js_path` so the same cartridge consistently routes to the same worker,
10+
maximising the module cache hit rate inside the Deno process.
11+
12+
Pool size defaults to 5 (configurable via `:js_worker_pool_size` in the
13+
`:boj_rest` application env).
14+
15+
If Deno or the pool worker script is absent, the pool starts with 0 workers
16+
(instead of crashing) and `BojRest.JsInvoker.invoke/4` falls back to the
17+
fork-per-call path automatically.
18+
"""
19+
use Supervisor
20+
require Logger
21+
22+
@default_size 5
23+
24+
# ── public API ───────────────────────────────────────────────────────────────
25+
26+
def start_link(opts), do: Supervisor.start_link(__MODULE__, opts, name: __MODULE__)
27+
28+
@doc """
29+
Return the pool size that was configured at startup, or 0 if the pool is empty.
30+
"""
31+
@spec pool_size() :: non_neg_integer()
32+
def pool_size do
33+
Application.get_env(:boj_rest, :js_worker_pool_size, @default_size)
34+
end
35+
36+
@doc """
37+
Invoke `tool_name` on `mod_js_path` via the pool.
38+
39+
Uses consistent hashing on `mod_js_path` to select the same worker each time,
40+
maximising Deno module-cache locality. Falls back gracefully to
41+
`BojRest.JsInvoker.invoke/4` (fork-per-call) when the pool is not running.
42+
"""
43+
@spec invoke(String.t(), String.t(), map(), map()) ::
44+
{:ok, map()} | {:error, map()}
45+
def invoke(mod_js_path, tool_name, args, extra_env \\ %{}) do
46+
case pick_worker(mod_js_path) do
47+
nil ->
48+
BojRest.JsInvoker.invoke(mod_js_path, tool_name, args, extra_env)
49+
50+
worker ->
51+
BojRest.JsWorker.invoke(worker, mod_js_path, tool_name, args, extra_env)
52+
end
53+
end
54+
55+
# ── Supervisor callbacks ────────────────────────────────────���───────────────
56+
57+
@impl true
58+
def init(opts) do
59+
deno = BojRest.JsInvoker.deno_path()
60+
runner = runner_path()
61+
size = Keyword.get(opts, :pool_size, Application.get_env(:boj_rest, :js_worker_pool_size, @default_size))
62+
63+
if is_nil(deno) or is_nil(runner) do
64+
Logger.warning("JsWorkerPool: Deno binary or pool worker script not found — starting with 0 workers; JS invocations will use fork-per-call fallback")
65+
Supervisor.init([], strategy: :one_for_one)
66+
else
67+
workers =
68+
for i <- 0..(size - 1) do
69+
name = worker_name(i)
70+
71+
%{
72+
id: name,
73+
start: {BojRest.JsWorker, :start_link, [[name: name, deno_path: deno, runner_path: runner]]},
74+
restart: :permanent
75+
}
76+
end
77+
78+
Logger.info("JsWorkerPool: starting #{size} workers", deno: deno)
79+
Supervisor.init(workers, strategy: :one_for_one)
80+
end
81+
end
82+
83+
# ── private ────────────────────────────────────────────────────────────��───
84+
85+
defp pick_worker(mod_js_path) do
86+
size = Application.get_env(:boj_rest, :js_worker_pool_size, @default_size)
87+
idx = :erlang.phash2(mod_js_path, size)
88+
name = worker_name(idx)
89+
90+
case Process.whereis(name) do
91+
nil -> nil
92+
_pid -> name
93+
end
94+
end
95+
96+
defp worker_name(i), do: :"boj_js_worker_#{i}"
97+
98+
defp runner_path do
99+
priv =
100+
case :code.priv_dir(:boj_rest) do
101+
{:error, _} -> nil
102+
dir -> Path.join(to_string(dir), "js_pool_worker.js")
103+
end
104+
105+
dev_fallback =
106+
__DIR__
107+
|> Path.join("../../../priv/js_pool_worker.js")
108+
|> Path.expand()
109+
110+
cond do
111+
is_binary(priv) and File.regular?(priv) -> priv
112+
File.regular?(dev_fallback) -> dev_fallback
113+
true -> nil
114+
end
115+
end
116+
end

elixir/lib/boj_rest/router.ex

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,8 @@ defmodule BojRest.Router do
121121
if Map.has_key?(cart, "ffi") do
122122
BojRest.Invoker.invoke(cartridge_so_path(cart), tool, args, creds)
123123
else
124-
BojRest.JsInvoker.invoke(cartridge_mod_path(cart), tool, args, creds)
124+
# JsWorkerPool.invoke falls back to JsInvoker.invoke when pool is absent.
125+
BojRest.JsWorkerPool.invoke(cartridge_mod_path(cart), tool, args, creds)
125126
end
126127
end
127128

elixir/priv/js_pool_worker.js

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
// SPDX-License-Identifier: PMPL-1.0-or-later
2+
// Copyright (c) 2026 Jonathan D.A. Jewell (hyperpolymath) <j.d.a.jewell@open.ac.uk>
3+
//
4+
// Long-lived Deno pool worker for BojRest.JsWorkerPool.
5+
// Reads newline-delimited JSON requests from stdin, writes responses to stdout.
6+
//
7+
// Request: {"id":"...","mod":"/path/to/mod.js","tool":"toolName","args":{...},"env":{...}}
8+
// Response: {"id":"...","status":200,"data":{...}}
9+
//
10+
// Requests are processed sequentially (one at a time via await). The "env"
11+
// field, when present, injects per-invocation credential vars into Deno.env
12+
// before calling handleTool and cleans them up afterward — safe because there
13+
// is no request concurrency in a single worker.
14+
//
15+
// Module-level caching: once a mod.js is imported, its handleTool function is
16+
// cached for the lifetime of the worker. This eliminates the re-import cost
17+
// on every call after the first (the main point of the pool).
18+
19+
const decoder = new TextDecoder();
20+
const encoder = new TextEncoder();
21+
/** @type {Map<string, Function>} */
22+
const modCache = new Map();
23+
24+
let buffer = "";
25+
26+
/**
27+
* @param {{ id: string, mod: string, tool: string, args?: object, env?: Record<string,string> }} req
28+
*/
29+
async function processRequest(req) {
30+
const { id, mod, tool, args = {}, env = {} } = req;
31+
32+
const envKeys = Object.keys(env);
33+
for (const [k, v] of Object.entries(env)) Deno.env.set(k, v);
34+
35+
try {
36+
let handleTool = modCache.get(mod);
37+
if (!handleTool) {
38+
const modUrl = mod.startsWith("/") ? `file://${mod}` : mod;
39+
const imported = await import(modUrl);
40+
handleTool = imported.handleTool;
41+
modCache.set(mod, handleTool);
42+
}
43+
44+
const result = await handleTool(tool, args);
45+
const out = JSON.stringify({ id, status: result?.status ?? 200, data: result?.data ?? result });
46+
await Deno.stdout.write(encoder.encode(out + "\n"));
47+
} catch (err) {
48+
const out = JSON.stringify({ id, status: 500, data: { error: String(err) } });
49+
await Deno.stdout.write(encoder.encode(out + "\n"));
50+
} finally {
51+
for (const k of envKeys) Deno.env.delete(k);
52+
}
53+
}
54+
55+
for await (const chunk of Deno.stdin.readable) {
56+
buffer += decoder.decode(chunk);
57+
let nl;
58+
while ((nl = buffer.indexOf("\n")) !== -1) {
59+
const line = buffer.slice(0, nl).trim();
60+
buffer = buffer.slice(nl + 1);
61+
if (!line) continue;
62+
let req;
63+
try {
64+
req = JSON.parse(line);
65+
} catch {
66+
// malformed — skip
67+
continue;
68+
}
69+
await processRequest(req);
70+
}
71+
}

0 commit comments

Comments
 (0)