OpenAI API Instrumentation (e.g. seed, logprobs, echo, an array prompt)

[monotophic]

Authored by Claude Opus in Claude Code, analysis in partnership with @monotophic

The OpenAI-compatible server refuses several ordinary request fields outright
today, and the pipe it speaks to the engine over is checked in neither
direction. This changes both. It is six files — Python, one JSON test fixture,
and documentation. No engine source, no build change.

seed is accepted and ignored rather than rejected. No engine and no field
on the wire protocol reads a per-request seed, so the value genuinely has no
effect at any temperature; docs/api.md says exactly that instead of implying a
determinism this build cannot deliver.

Per-token log probabilities and prompt echo are served on the glm engine.
The engine on dev already implements the numeric channel — c/decode_batch.h
parses the logprobs=k and ids=1 SUBMIT extensions and c/colibri.c already
emits the per-token tail on DATA/ECHO frames. Only the server side was
missing. /v1/completions now takes the legacy integer logprobs (0–32) and
boolean echo; /v1/chat/completions takes boolean logprobs with
top_logprobs. Every other engine refuses the fields with a named 400 rather
than ignoring them, and an engine build older than the SUBMIT extension is
answered with a named 503 after a bounded wait instead of leaving the request to
hang. The costs are documented rather than hidden: opting in forfeits
prefix-cache reuse for that request, the top-k table is unsorted on the wire,
and the sampled token is not guaranteed to appear in its own table.

prompt accepts arrays. A flat array of token ids is one pre-tokenized
prompt; an array of strings or an array of token-id arrays is a batch — the
shape unmodified lm-eval sends for tokenized loglikelihood requests, which
dev rejects. A batch is all-or-nothing: every member's shape is validated
before the first engine submit, members are submitted one at a time each with
its own UTF-8 decoder and its own echo reassembly, any member's failure fails
the whole request with that member named as prompt[i], and nothing reaches the
client until every member has finished. A member cap and two token budgets bound
what one request can hold in memory. group_score — a future opt-in that would
change the response shape — is refused with a named 400 rather than silently
ignored, because ignoring it would answer a different question than the client
asked.

The server↔engine protocol is fail-closed in both directions. Every
SUBMIT/CANCEL/STOP write is checked, so a write onto a dead engine becomes
a named HTTP 500 instead of a silent connection close; upstream's frame-writer
structure and its IMAGE-frame ordering are untouched — the checking is
expressed inside them. In the other direction the dispatcher validates frame
grammar rather than desynchronizing, and it drains four group-scoring frame
kinds (GRPP/GRPG/GRPS/GRPE) so that a server built from this branch stays
correct if it is ever paired with an engine that emits them. No engine in this
tree emits them (verified by grep over the whole repository at this head: the
only hits are this server, its tests, and docs/api.md); the engine that would
is a group-scoring channel proposed separately, and nothing in this PR waits for
it.

One default-path behaviour is changed deliberately: /v1/messages streaming now
commits its HTTP 200 on the engine's ACCEPT rather than at scheduler
admission, matching what the OpenAI-style endpoints already did. A refusal
discovered before acceptance is now a real HTTP 400 in the Anthropic error
envelope instead of a committed 200 followed by a truncated event stream. On a
healthy engine the framing and event order are unchanged; the consequence to
know is that a request queued behind another generation now waits silently,
exactly as the OpenAI-style path already made it wait.

Context. This PR is one of a set of independent contributions derived from a
single locally-verified working tree: fp8 container support (already proposed as
#1102), this server API work, four evidence-tooling and engine-instrumentation
contributions, and a group-scoring serve channel proposed later. This one
carries the OpenAI-compatible server's request surfaces and its engine-protocol
contract. It stands alone — measured, not assumed: its branch sits directly on
dev at e1efc68 (its merge base with dev is dev itself), its whole suite
is green there with nothing else from the set present (make check, 908 tests),
and merging or declining the others does not affect it; separately, the three
programs finished so far were merged together locally and the combined suite is
green (1244 tests), so the set is also known to compose. The others are proposed
separately, each with its own evidence.

Behavioral contract

Everything that worked before still works.

• A request that uses none of the new fields produces a byte-identical SUBMIT
  header and a byte-identical response body to dev's.
• Every frame the server writes to the engine — SUBMIT, SUBMIT preceded by
  IMAGE, CANCEL, STOP — is byte-identical to dev's, in the same order,
  under the same single lock acquisition.
• Every test method that exists on dev still exists here, in both server test
  modules; none was removed to make room for a new behaviour.
• The one intended exception is the /v1/messages streaming commit point,
  described above.

seed

• A request carrying seed on /v1/completions or /v1/chat/completions is
  processed exactly as the same request without it.
• The value reaches no wire frame and no engine, at any temperature.

Log probabilities and prompt echo

• /v1/completions accepts integer logprobs 0–32 and boolean echo; 0,
  false and null all mean "no log probabilities" and return
  choices[].logprobs: null with a 200, while boolean true is a named 400.
• /v1/chat/completions accepts boolean logprobs plus integer top_logprobs
  (0–32) and returns choices[].logprobs.content[]; any integer logprobs is a
  named 400, and echo is refused.
• top_logprobs is type- and range-checked on chat even when logprobs is
  false or absent; a non-boolean echo is a named 400 on both endpoints.
• On any engine that does not implement the channel, a logprobs request is a
  named 400 — never a silent no-op.
• logprobs together with stream is a named 400.
• An engine build that never acknowledges an opted-in SUBMIT produces a named
  503 within COLI_LOGPROBS_ACCEPT_TIMEOUT (default 30 s), never a hang; the
  request is treated as cancelled rather than left pending.
• Non-finite values serialize as JSON null, never a clamped number. Echo
  positions are placed by the wire's own pos field, not arrival order.
  text_offset is a character offset into the returned text, counted from 0.
  The chosen token's log probability is read from its own record, not looked up
  by rank in the top-k table.

Array and token-id prompts on /v1/completions

• A single-member array — one string, or one token-id list, or a nested
  batch-of-one — produces exactly what the same prompt produces as a flat
  single-prompt request.
• An array must be non-empty and homogeneous (all strings, all token ids, or all
  token-id arrays), at any size; each violation is a named 400 on prompt.
• Token-id prompts, flat or nested, are glm-engine-only; every other engine
  returns a named 400 rather than mis-tokenizing decimal digits as text.
• A real batch returns one choice per member, choices[i].index == i, in order,
  with usage summed across members.
• Every member's shape is validated before the first engine submit, so a
  malformed member N is a clean 400 naming prompt[N-1] with zero submits made.
• A member the engine itself rejects after earlier members were submitted is
  still attributed to that member; a client-fault error keeps the engine's own
  status, code and param with only the message gaining the index; an engine
  failure the server cannot name is one un-attributed 500 for the whole batch.
• Nothing is written to the client until every member finishes; a client that
  disconnects mid-batch stops it at the member in flight.
• One member's trailing partial UTF-8 character, or its logprobs table, can
  never leak into another member's text.
• A batch is capped at 128 members; at 65,536 total prompt tokens (token-id
  members counted in tokens, string members in UTF-8 bytes as an upper bound);
  and, on the generated side, at members × the request's effective max_tokens
  not exceeding 65,536, checked before any engine submit. A length-1 array is
  exempt from both budgets and keeps the flat path's own oversize handling. The
  scheduler admission is released before the response is written.
• stream: true and n other than 1 are refused with a named 400 on a batch.
• group_score present and neither null nor false is a named 400
  (code: "unsupported_value") on /v1/completions, flat or array, before any
  prompt intake or engine submit. /v1/chat/completions and /v1/messages do
  not read the field.

Engine protocol

• Every server→engine frame write is checked; a failed write is a named HTTP 500
  engine_error while the response is uncommitted, and ends the stream once it
  is committed — never a silent close.
• GRPP/GRPG payloads are drained under the same size bound and terminator
  check as DATA, and GRPS/GRPE are consumed with no effect; nothing from
  them reaches any request's event queue. A malformed group frame raises the
  dispatcher's existing named error.
• Every other frame kind the dispatcher handled on dev is handled identically.
• The engine child runs under the default POSIX SIGPIPE disposition, so a
  disconnected reader terminates it at its next write with no evidence records
  after the failure point.

Capstone matrix

One decisive artifact per claim; each would fail if the claim were false. Test
citations are file:line at this branch's head.

claim	decisive evidence
seed reaches no wire frame	c/tests/test_openai_server.py:2209 test_seed_accepted_and_absent_from_submit_frame — the SUBMIT frame bytes for a seeded request are compared against the unseeded request's; any leak of the value changes them
requests that use none of the new fields are unchanged	two byte-level pins: c/tests/test_openai_server.py:4305 asserts the exact SUBMIT header literal, and :4418 asserts the plain completions response's field set and values
no frame byte the server writes changed	c/tests/test_openai_server.py:6198 test_wire_transcript_is_byte_identical_to_base — the whole SUBMIT / IMAGE+SUBMIT / CANCEL / STOP sequence is compared to a literal captured transcript, so a reorder, a missing flush boundary or one stray byte fails it
logprobs are refused, never ignored, off the glm engine	c/tests/test_openai_server.py:4251 test_break_it_logprobs_rejected_for_non_glm_engine
an old engine cannot hang an opted-in request	c/tests/test_openai_server.py:4703 test_old_engine_rejection_times_out_with_a_named_503 — it runs generate() on its own thread precisely so that a missing deadline fails this one test instead of hanging the suite
the documented environment variable is the one actually read	c/tests/test_openai_server.py:4831 and :4834 import the module in a fresh subprocess with the real variable name set and unset, so a typo in the name or a changed default cannot ship green
a batch of one is the flat path	c/tests/test_openai_server.py:4093 test_nested_batch_of_one_prompt_is_identical_to_flat
a rejected batch does no engine work	c/tests/test_openai_server.py:5224 test_rejected_batch_makes_no_engine_submits — the engine's call count is unchanged across three differently-malformed requests
a member the engine rejects mid-batch is still named	c/tests/test_openai_server.py:5256 test_context_exceeded_mid_batch_names_member_in_param — param: "prompt[1]" with the engine's own code: "context_length_exceeded" preserved
each batch choice equals its single-prompt twin (server logic)	c/tests/test_openai_server.py:5502 and :5486 compare every batched choice, field by field, against the same prompt sent alone — strings and token ids
each batch choice equals its single-prompt twin (real engine, one shared KV slot)	a real glm engine and a real container on a Linux CPU host, one KV slot, --kv-slots 1: the same three prompts were sent as singles, as a batch, as a reversed batch, as singles again, and as a duplicate-member batch. Every (text, finish_reason) was identical across all five arms, and the batch's aggregate completion_tokens equalled the sum of the singles'. The prefix cache was live and demonstrably reached — the first request logged prefix 0/68 token, prefill 68 and later ones logged prefix 68/68 … prefill 0 and prefix 40/68 … prefill 28 — so identity holds with another member's residue in the slot, not because the slot was never warm. All 20 requests landed on slot 0, and each batch was one engine admission. Corroborating the "no engine change" claim: the engine built from this branch's own tree is sha256-identical to the engine built from dev.
group_score fails closed	c/tests/test_openai_server.py:5157 test_group_score_opt_in_is_refused_fail_closed, on both the flat and the array request shape
the group frame kinds cannot desync the pipe	c/tests/test_openai_server.py:2007 test_group_frames_between_data_frames_drain_leaving_second_data_intact — each kind is interleaved between a request's own two DATA frames and the second must still arrive byte-for-byte
a token-id array prompt is budgeted by its real token count	c/tests/test_openai_server.py:4199 test_array_prompt_token_budget_counts_actual_tokens_for_token_id_batches — an over-budget token-id batch gets the documented refusal and the engine sees no submit
a batch that names no cache_slot runs every member on the one slot the scheduler picked	c/tests/test_openai_server.py:5804 test_batch_without_cache_slot_shares_the_one_slot_the_scheduler_picked — every recorded generate call carries the same slot
a failed CANCEL/STOP write reaching an already-committed stream ends the stream, never splices a 500	c/tests/test_openai_server.py:3822 KeepAliveFramingTest.test_write_failure_reaching_the_committed_stream_ends_it_cleanly — a real Engine + fake subprocess ACCEPTs, streams partial text, then fails the STOP write; exactly one HTTP status line reaches the client, the pre-failure text survives, no [DONE] and no error body are spliced in, the connection closes, and the failure is logged
Anthropic /v1/messages streaming sends no bytes at all until the engine's first accept-equivalent frame, then delivers the deferred 200 and full SSE sequence unchanged	c/tests/test_openai_server.py:3617 AnthropicColdPrefillTest.test_no_bytes_reach_the_client_until_the_engine_finally_accepts — with the engine held open before on_accept, the raw socket yields a bounded read timeout (zero bytes); once released, HTTP/1.1 200 plus message_start…message_stop arrive intact
a dead engine is a named 500, not silence	c/tests/test_openai_server.py:5877 test_dead_engine_submit_is_a_named_500_engine_error_not_silence — at the pre-fix revision the BrokenPipeError fell into the client-hangup handler and the client saw a silent close
the SIGPIPE policy is real, not merely asserted	c/tests/test_openai_server.py:6249 — a real fork/pipe child under SIG_DFL is killed by signal 13 at its next write with no completion record after the failure point, and :6267 pins the precondition by asserting the server passes no restore_signals kwarg at all
the Anthropic stream no longer commits before acceptance	c/tests/test_openai_server.py:3464 gets a real 400 in the Anthropic envelope with zero SSE bytes where dev sent a 200; :3521 pins the healthy-engine framing and exact event order as unchanged

Fuller matrix, disclosures, known gaps, origin accounting, and namespace claims

Requirement → instrument → result

requirement	instrument	result
whole suite green at this head	make check (clean + portable C build + C suite + Python suite)	Ran 908 tests … OK (skipped=42), 4m39s
the server module is not flaky	test_openai_server.py run three more times	336 tests OK each time, 97–99 s, zero flake
every commit stands on its own	per-commit test run and per-commit changed-line lint for all eight commits	green each; 0 lint findings each
the new suite is not self-referential	the boundary tests read constants back from the module, but the constants themselves are pinned to literals elsewhere — LOGPROBS_TOP_K_CAP == 32 at :573, PROMPT_BATCH_CAP == 128 at :4127, both 65,536 budgets at :4161 and :5596	a constant mutation cannot silently widen the boundary tests along with themselves
the engine's numeric grammar is the shipped one	the parser and the test fixtures accept the %.6f form the engine on dev prints, with non-dyadic values so a different grammar is distinguishable	accepted; nan/inf/-inf parse and serialize as JSON null
assembly helpers place echo positions correctly	multibyte and split-codepoint offset fixtures with literal expected values (:702, :806)	exact character offsets, monotonic, trailing incomplete sequence flushed
batch member isolation	a codepoint deliberately split across two members (:5560), and hostile echo positions mid-batch (:5572)	per-member decoders hold; the hostile case fails clean, bounded, never hangs
batch partial output is impossible	:5233 mid-batch failure, :5346 client disconnect mid-batch	no partial response; the disconnect stops further submits
the SIGPIPE claim's wording	docs/api.md says "signal 13 (wait status 141)"; the test asserts the child's return code is -signal.SIGPIPE	equivalent (a shell reports a signal-13 death as 141); the literal 141 is not itself asserted anywhere
suite runs on a clean upstream checkout	every fixture in the new tests is self-contained; the POSIX-only fork/pipe class is skipped by name with a reason on Windows	no test depends on anything outside the repository
the whole set composes	the three finished programs merged together locally and the combined suite run	make check exit 0, 1244 tests OK (45 skipped) — verified together locally, on top of dev at e1efc68

Disclosures

• The GRPP/GRPG/GRPS/GRPE drain arms have no emitter in this tree.
  A repository-wide grep at this head finds them only in c/openai_server.py,
  its test module, and docs/api.md — no engine source, no header. They are
  drained, never surfaced: nothing from them reaches any request. The engine
  that emits them is a group-scoring serve channel proposed separately; if you
  would rather not carry protocol tolerance ahead of its producer, the commit
  that adds them (feat(api): drain group-scoring frames a future engine may emit) is self-contained and can be dropped from this PR without touching
  anything else.
• /v1/messages streaming first-byte timing. This is the one default-path
  behaviour change, and it is intended. With the in-tree engine the delay is one
  SUBMIT→ACCEPT round trip, because the engine emits ACCEPT before prefill
  and polls its stdin on every decode step. Against an engine binary old enough
  never to send ACCEPT at all, the first _accept fires on the first DATA
  or DONE instead, so a cold multi-minute prefill would send zero bytes where
  dev sent message_start plus a periodic ping — a client with a short idle
  timeout could newly time out. That mismatched pairing is not covered by a
  test; see the gaps below. The change also creates a genuine new expectation
  on the Engine contract: a generate implementation that never calls
  on_accept now yields no Anthropic SSE body at all. Production cannot reach
  that (_accept is guaranteed before the first decode), but any in-tree test
  double must model the ACCEPT frame, and the two-line change to
  c/tests/test_anthropic_messages.py in this PR is exactly that.
• Environment variable and timeout documentation. This PR introduces one
  environment variable, COLI_LOGPROBS_ACCEPT_TIMEOUT (default 30 seconds),
  which bounds how long an opted-in request waits for the engine's
  acknowledgement before being cancelled and answered with a named 503. It is
  documented in docs/api.md and pinned by its real name and its default in the
  suite rather than only through the module attribute the rest of the tests
  patch. The other public knobs this PR documents — the 32-entry top-k cap, the
  128-member batch cap, and the two 65,536 budgets — are module constants, not
  environment variables, and each is pinned to its literal value by a test.
• Suite cost, before you measure it yourselves. test_openai_server.py goes
  from 161 to 336 test methods and is now roughly 96 seconds of the Python
  suite's ~140. That is the dominant cost of this change; the C suite and the
  build are untouched.

Known gaps, stated plainly

None remain open. The four items an earlier draft listed here — token-id batch budgeting, cache_slot on a batch, a
write failure reaching an already-committed stream, and Anthropic streaming against an engine that never accepts — are
each pinned by a test named in the capstone matrix above (the last two added at the head of this branch).

Origin accounting

The base is unmoved: this branch's merge base with dev is dev itself, so
no change here is caused by upstream motion. By commit:

commit	origin class
feat(api): accept and discard per-request seed (documented no-op)	re-expressed from our own earlier work on this surface. Adjacent consequence, disclosed in the commit: the golden-fixture capture battery's err_seed case is renamed seed_accepted, because the case now captures a 200.
fix(api): defer the Anthropic streaming 200 until the engine accepts the prompt	re-expressed from our own earlier work. Carries the new Engine expectation described above and the two-line ACCEPT simulation the in-tree Anthropic test double needed.
feat(api): serve OpenAI-style logprobs and prompt echo on the glm engine	re-expressed, with three tightenings found by our own re-review before proposing: top_logprobs is validated on chat even when logprobs is off (it was silently ignored), echo's boolean check is the same on both endpoints, and the top-k cap is pinned by its literal value rather than by the symbol it polices. Adjacent consequence: the capture battery's err_logprobs case becomes logprobs_served.
feat(api): accept array and token-id prompts on /v1/completions	re-expressed. Two documentation claims this branch had written about its own behaviour were wrong and were corrected here before proposing: the enumerated list of code values for param: "prompt[i]" was presented as closed when the engine's own mid-batch context_length_exceeded is preserved through the member rewrite (the branch's own passing test asserts it), and the multi-prompt section claimed stream: true was refused "the same as everywhere else on this endpoint" when the flat path refuses only the narrower logprobs-plus-stream combination and names a different param. Both now say what the code does, including the param divergence a client keying off error.param would hit. Adjacent consequence: the capture battery's err_array_prompt case becomes array_prompt_token_ids, because this PR moves that case from a 400 to a 200 on the glm engine — the third of the three renames.
fix(api): fail closed on an unsupported group_score opt-in	re-expressed.
feat(api): drain group-scoring frames a future engine may emit	new content, serving a channel proposed separately; disclosed above, and separable from the rest of this PR.
fix(api): check every server->engine frame write	re-expressed inside upstream's existing frame-writer structure rather than replacing it: the SUBMIT block keeps its shape and its IMAGE-frame ordering under one lock acquisition and gains only a not-running check, a flush, and an OSError→named-RuntimeError conversion. The five CANCEL/STOP sites, which were five identical inline blocks, call one small helper; that helper is deliberately not used by SUBMIT.
fix(api): drop Engine.generate()'s pending-map entry on every exit	a pre-existing defect on dev, found by re-review of this branch, not one this branch introduces. dev's generate() has the same pending map with no finally: the entry is popped only on a SUBMIT-write failure and by the dispatcher's own DONE/ERROR handling, so a write failure on the CANCEL/STOP path leaves it behind. Bounded on dev — that error means the engine's stdin is broken, so the dispatcher hits EOF and clears the map — but it should not depend on that. Note for review: this commit's diff looks large and is almost entirely re-indentation; ignoring whitespace it is 11 changed lines in c/openai_server.py plus one test.

Style note: changed lines were held to the file's measured local idiom via a
diff-scoped consistency check; no surrounding code was reformatted.

Shared-namespace claims

This PR claims the following names in namespaces other contributions could
collide with. The list is scanned against dev HEAD and against every open pull
request's diff immediately before push, and re-scanned if the push slips.

• Environment variable: COLI_LOGPROBS_ACCEPT_TIMEOUT (one; new).
• API error code strings: prompt_batch_cap_exceeded,
  prompt_batch_token_budget_exceeded, batch_completion_budget_exceeded,
  engine_tok_ids_unsupported (four; new). Existing codes reused unchanged:
  invalid_value, unsupported_value, unsupported_parameter,
  context_length_exceeded, engine_error.
• Wire protocol: nothing new is claimed. The SUBMIT extension keys this
  server now sends, logprobs=k and ids=1, are already defined and parsed by
  the engine on dev (c/decode_batch.h); this PR adds a sender, not a key.
  The four GRP* frame kinds are consumed only — nothing here emits them.
• No new command-line flags, no file magic, no format ordinals.

Durable vs current state: the request contracts, the error codes, the
protocol rules and the documented limits are durable. The counts (908 tests at
this head, 336 in the server module, 1244 for the three programs merged
together) and the fleet run quoted above are current state, measured 2026-09-05
against base e1efc68 on macOS (Apple silicon) and, for the KV-slot cell, on a
Linux CPU host with GCC.