Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions docs/api-reference/endpoints.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,34 @@ On the response, the gateway adds `X-Gateway-Provider` (the name of the
provider that served the request) and scans non-2xx bodies to redact any
echoed credential before it reaches the client.

## Attribution headers

Every routed surface — `/v1/chat/completions` (streamed or not),
`/v1/completions`, `/v1/embeddings`, `/v1/images/generations`, `/v1/rerank`,
`/v1/moderations`, `/v1/audio/transcriptions`, `/v1/audio/translations` and
`/v1/audio/speech` — answers with four headers naming the target that served
it, or on failure the last one attempted:

| Header | Value |
|---|---|
| `X-Gateway-Provider` | the serving target's canonical provider (`openai`) |
| `X-Gateway-Target` | the target key as configured: `targets[].virtual_key` |
| `X-Gateway-Model` | the upstream model sent to the provider, after `model_map` |
| `X-Gateway-Attempts` | routing-layer attempts for the request: provider calls plus local breaker or concurrency refusals, retries and failovers included |

On a stream they are written before the first chunk. A request refused before
any target was attempted — a plugin denial, a model nothing serves — carries
none. The value is never a credential: the target key is the config string,
not the key it names. The pass-through proxy above emits
`X-Gateway-Provider` only.

One request header goes the other way. `X-Gateway-Metadata`, a JSON object of
at most 32 string, number or boolean values within 4 KiB, is the single
request header [conditional routing](/routing/conditional) may read
(`key: metadata`, `field: <entry>`), on `/v1/chat/completions` and
`/v1/completions`. It never reaches a provider, no other header is exposed to
a rule, and a malformed value is the caller's `400`.

An upstream connection failure surfaces as `502 upstream_error`. A
before-request content guardrail configured on the deployment that cannot
read the request body (a multipart upload, binary audio, or anything not
Expand Down
2 changes: 1 addition & 1 deletion docs/api-reference/streaming.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The gateway needs real token counts for metering, cost accounting, and the budge

## Mid-stream errors

If the upstream provider fails *after* the stream has started, the gateway cannot reuse a normal HTTP status code (headers are already sent). Instead it emits a single error event as a `data:` line and then closes the stream. No `[DONE]` sentinel follows an error event. The failure is also recorded on the request's span, counted in Prometheus metrics, and written to the request log via the `on_error` plugin stage — a mid-stream failure is no longer a silent truncation on any of those surfaces.
The response headers — including the `X-Gateway-Provider`, `X-Gateway-Target`, `X-Gateway-Model` and `X-Gateway-Attempts` [attribution headers](/api-reference/endpoints#attribution-headers) — are sent before the first chunk, because the routing walk has finished choosing by then. If the upstream provider fails *after* the stream has started, the gateway cannot reuse a normal HTTP status code (headers are already sent). Instead it emits a single error event as a `data:` line and then closes the stream. No `[DONE]` sentinel follows an error event. The failure is also recorded on the request's span, counted in Prometheus metrics, and written to the request log via the `on_error` plugin stage — a mid-stream failure is no longer a silent truncation on any of those surfaces.

The error chunk has this exact JSON shape:

Expand Down
102 changes: 99 additions & 3 deletions docs/changelog.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,106 @@
---
title: Changelog
description: "Release history for Ferro Labs AI Gateway v0.4.5 through v1.4.1: the v1.4.0 breaking routing pipeline, embedded dashboard, and native rerank/audio endpoints."
keywords: [AI gateway changelog, LLM proxy releases, AI gateway version history, AI gateway v1.4.1, AI gateway v1.4.0 breaking changes, embedded dashboard, unified routing pipeline, MCP stdio transport, OpenTelemetry tracing, gateway updates]
description: "Release history for Ferro Labs AI Gateway v0.4.5 through v1.5.2: the v1.5.2 routing depth release, the v1.5.1 routing correction, and the importable v1.5.0 runtime."
keywords: [AI gateway changelog, LLM proxy releases, AI gateway version history, AI gateway v1.5.2, sticky routing, target_keys, attribution headers, model_map, failover-safe routing, AI gateway v1.4.0 breaking changes, embedded dashboard, unified routing pipeline, MCP stdio transport, OpenTelemetry tracing, gateway updates]
---

Full release notes are also on [GitHub Releases](https://github.com/ferro-labs/ai-gateway/releases). **v1.4.1 is the latest tag.**
Full release notes are also on [GitHub Releases](https://github.com/ferro-labs/ai-gateway/releases). **v1.5.2 is the latest tag.**

## v1.5.2 — 2026-09-03 — Routing depth

Every routed surface now ranks through one strategy implementation, and the routing layer gains the operator controls a production gateway needs: per-attempt timeouts, a `429` cooldown, a typed context-length failover class, sticky sessions, rule target chains, bounded conditional predicates, attribution headers on every response, and a host-supplied price catalog. Everything ships in the open-source gateway; no configuration key is removed, and the Go API only grows. See [Routing](/routing) for the updated contract.

### What's new in v1.5.2

- **Attribution headers on every routed surface** — `X-Gateway-Provider`, `X-Gateway-Target`, `X-Gateway-Model` and `X-Gateway-Attempts` on chat (before a stream's first chunk), legacy completions, embeddings, images, rerank, moderations, transcriptions, translations and speech; a failed request names the last target attempted. `ferro.routing.attempt` is emitted on the request span. See [Attribution headers](/api-reference/endpoints#attribution-headers).
- **`targets[].timeout`** bounds one physical attempt inside `request_timeout`, so a hung primary no longer consumes the whole request budget before a pool mode moves on. Streams are bounded only until the provider answers.
- **`429` cooldown** — a target that answers `429` is parked for its `Retry-After` (five seconds when absent, a minute at most) so the next request does not pay another `429` on it. Process-local; the breaker is untouched.
- **Typed context-length failover** — a provider's own statement that the prompt exceeded its context window (the OpenAI-compatible, Anthropic and Gemini envelopes) fails over to a sibling, whose model may have a larger window. Every other `4xx` still stops.
- **`strategy.sticky: { on: user, ttl: "1h" }`** under `loadbalance` and `ab-test` pins a `user` to one target or variant with a stateless hash. See [Load balance](/routing/loadbalance#sticky-sessions).
- **`target_keys: [a, b]`** on `conditions[]` and `content_conditions[]` names an ordered chain for a rule, walked on failover-safe failures and never left. See [Conditional](/routing/conditional).
- **Conditional predicates** `user`, `stream`, `has_tools`, and `metadata` + `field` reading the single `X-Gateway-Metadata` request header.
- **`strategy.failover_on_status_codes`** adds upstream statuses to the failover-safe set; `400`, `401`, `403`, `404` and `422` cannot be listed.
- **`aigateway.WithCatalog(models.Catalog)`** hands an embedding host's own price catalog to the gateway in place of the embedded or remote one; `aigateway.WithRoutingAttribution` reads the attribution back.
- **Embedded dashboard** — the strategy panel shows `model_map`, rule chains and the new predicates.

### Behaviour changes in v1.5.2

- **A rule that names one target is exact.** Under `conditional` and `content-based`, an open circuit on the matched target used to borrow a healthy sibling; it now answers `503`, and a rule that wants a stand-in lists one in `target_keys`. On non-chat surfaces `content-based` routes to the first target that can serve the request, alone.
- **One ranker for every surface.** Embeddings, images, rerank, moderation, transcription and speech previously ranked through a second implementation that differed from chat in its random source, unseen-latency order, cost input and unpriced placement. The same config and health now produce the same candidate order everywhere.
- **`least-latency` keeps learning.** Samples are keyed by target and upstream model, expire after five minutes, and one request in ten leads with a sampled runner-up so the leader cannot lock in. A stream's sample is its time to first chunk rather than its whole drain.
- **`cost-optimized` prices input plus output** — the request's completion ceiling or 256 tokens — at the catalog rate for the model's mode on every surface; equal-cost targets draw by `targets[].weight`, and a negative weight is refused.
- **Validation** — an `ab_variants[]` entry without a `label` no longer loads; a `single` strategy with more than one target logs a warning naming the unused targets.

---

## v1.5.1 — 2026-09-01 — Routing reliability correction

A patch that makes the routing strategies truthful about failure, gives one client-facing model name a different upstream ID per provider, and makes each physical attempt observable. Three behaviours an operator may notice change; no configuration keys are removed. See [Routing](/routing) for the updated contract.

### What's new in v1.5.1

- **`targets[].model_map`** — several providers can serve one visible model name while each receives its own upstream model ID (`smart: gpt-4o` on OpenAI, `smart: claude-sonnet-4-6` on Anthropic). Mapped names participate in routing and `/v1/models`; pricing uses the mapped ID. See [One model name, different upstream IDs](/routing#one-model-name-different-upstream-ids).
- **Attempt-level observability** — each provider call or local circuit-breaker/concurrency refusal can emit a `gateway.routing.attempt` event, retries and cross-target failovers included. Attempt events are opt-in per exporter and per custom observability provider; every existing exporter keeps receiving exactly one event per request. See [Observability](/guides/observability).
- **A/B attribution** — attempt and terminal events carry `ferro.routing.ab_variant_label`, and it stays the variant that was drawn through retries and failover. See [A/B test](/routing/ab-test).
- **Keyless strategy end-to-end suite** — `make test-e2e-strategies` exercises every routing mode over the real binary against three scriptable mock upstreams: failover classes, retries and `Retry-After`, breaker states, weight and variant distributions, `model_map` on unary and streamed requests, `/v1/models` and `/metrics` — with no provider credentials.
- **Install paths recorded** — the one-command installer (`get.ferrolabs.ai`), `ferrogw` on npm and PyPI, the Homebrew cask and Scoop manifest, and the GoReleaser platform archives landed on `main` during the v1.4.x line without a changelog entry, as did the README quickstart overhaul; both are now recorded. See [Install](/getting-started/install).

### Behaviour changes in v1.5.1

- **Pool modes fail over only after a failover-safe failure.** `fallback`, `loadbalance`, `least-latency`, `cost-optimized` and `ab-test` advance to another target after a transport failure, an attempt that timed out waiting on the target, `408`, `429`, `5xx`, an open circuit, or target saturation. They previously advanced after *any* failure, so a target answering `400`, `401`, `404` or `422` was silently covered by a sibling; those responses now reach the client. The request's own cancellation or deadline still stops routing, as does any provider-call failure under `single`, `conditional` and `content-based`.
- **The response `model` is the routed name** — the model the client asked for, after alias resolution — on every surface, streamed chunks included, instead of the identifier the provider reported. Provider calls, pricing and the `UpstreamModel` of an attempt event still use the mapped upstream model.
- **Ambiguous configurations are rejected at load** — duplicate target keys, an empty `targets[].virtual_key`, duplicate `ab_variants[].target_key` entries, and duplicated keys in a JSON config no longer load silently.
- **The embedded model catalog is parsed once per process.** Every gateway constructed without a reachable remote catalog previously decoded the 3 MB document again (about 90 ms); it now receives its own copy of the parsed catalog in about 3 ms.

### Fixed in v1.5.1

- A failure in an `after_request` plugin is timed and counted as a plugin failure and emits one failed terminal lifecycle event carrying the selected A/B variant, instead of ending without a duration sample or a terminal event.
- A stream whose upstream had already finished when the client hung up was recorded as a client cancellation about half the time; a completed and billed stream is now always recorded as completed.

---

## v1.5.0 — 2026-08-29 — The gateway is importable

A new public `run` package exposes the `ferrogw` program to Go code. `run.Main()` is what `cmd/ferrogw` now calls; `run.Run(ctx, opts...)` runs the same server under a caller-owned context and returns startup and listen errors instead of exiting the process, with context cancellation triggering the same graceful shutdown as `SIGTERM`. A custom binary is a `main` that blank-imports its plugins and calls `run.Main()` — the process lane. `httpgateway` (since v1.4.2) remains the library lane for mounting gateway surfaces behind your own middleware.

The server now binds its listener before it starts observing shutdown, so a cancellation that arrives during startup can no longer leave a listener behind. Existing `ferrogw` behaviour — commands, flags, exit codes — is unchanged.

---

## v1.4.5 — 2026-08-23 — Security patch: stdio MCP memory bound

- **A stdio MCP server can no longer exhaust gateway memory.** The stdio transport now applies the same 10 MiB bound as the HTTP transport, measured per JSON-RPC message, so an ordinary conversation of any length is unaffected. A previously working oversized tool result now fails — terminal for that server, not for one call: the transport closes and the registry withdraws the server and its tools. A server with a legitimate reason to return more than 10 MiB should page its results. See [MCP](/guides/mcp).
- **Three dependency advisories cleared** — `golang.org/x/text` v0.39.0, `golang.org/x/net` v0.56.0, `github.com/moby/go-archive` v0.3.0. None was reachable from gateway code; `govulncheck` reports zero vulnerabilities in every category.
- `SECURITY.md` names 1.4.x as the supported series.

No breaking changes to configuration or the API.

---

## v1.4.4 — 2026-08-18 — In-flight requests keep their provider price

An alias repointed to a different provider while a request was in flight could price that request against the replacement provider, even though the original served it. Routing now carries the pricing identity captured at provider selection through unary and streaming cost accounting. Attribution is unchanged: responses, metrics, spans and plugin context still name the routing alias.

---

## v1.4.3 — 2026-08-17 — Security patch: Go 1.25.13 and alias pricing

- **Go toolchain 1.25.13** — clears six standard-library advisories reachable from gateway code (`net/url`, `html/template`, `crypto/tls`, `net/http`, `encoding/xml`, `encoding/asn1`). No gateway code changes.
- **Dashboard toolchain** — `nanoid` 3.3.18 closes a high-severity advisory in a build-time dependency; nothing shipped in the embedded bundle was affected.
- **Registration aliases are priced correctly.** A provider registered under a routing alias (`RegisterProviderAs`, v1.4.2) was treated as unpriced by cost-optimized ranking and by streaming cost accounting; both now resolve the canonical provider for the catalog lookup. Deployments that register providers under their canonical name are unaffected.

No breaking changes.

---

## v1.4.2 — 2026-08-10 — Registration aliases, an embedding facade, and build provenance

- **`Gateway.RegisterProviderAs`** registers one provider under a distinct routing target, so a deployment can bind several credentials for the same canonical provider. The alias resolves every optional capability through the original provider — streaming, embeddings, images, rerank, moderation, audio, discovery, batch, Responses and pass-through.
- **`httpgateway` facade** — the Files/Batches, Responses and generic pass-through handlers are exposed to embedding applications, which keep their own authentication and tenant middleware while reusing the gateway's provider resolution, credential injection, governance and usage capture.
- **`GET /health` reports build provenance** — `version`, `commit` and `built` alongside provider status (`dev` / `none` / `unknown` for an unstamped local build).
- The configuration schema — `Config` and its sub-types, loader and validator — lives in the `config` package (`github.com/ferro-labs/ai-gateway/config`) since v1.4.0; for an embedder the migration is a one-line import.

---

## v1.4.1 — 2026-08-07 — Dependency security patch

Expand Down
2 changes: 1 addition & 1 deletion docs/faq/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ On routed surfaces — chat completions, streaming, embeddings, images — `X-Pr
<summary className={styles.q}>What happens if a provider is down?</summary>
<div className={styles.a}>

Depends on the strategy. Pool modes — `fallback`, `loadbalance`, `least-latency`, `cost-optimized`, `ab-test` — advance to the next target in the pool after a failover-safe failure — the provider was unreachable, timed out, returned `408`/`429`/`5xx`, or is circuit-open or saturated; any other `4xx` (`400`, `401`, `403`, `404`, `422`, …) is returned to the caller instead. Named modes — `single`, `conditional`, `content-based` — report the failure back to the caller instead, since something specifically chose that target. Every mode skips a target whose circuit breaker is open; if every target for a model is open, the request gets `503`. `targets[].retry` is honored under every mode, so set `attempts: 1` to keep single-attempt behavior even in a pool.
Depends on the strategy. Pool modes — `fallback`, `loadbalance`, `least-latency`, `cost-optimized`, `ab-test` — advance to the next target in the pool after a failover-safe failure — the provider was unreachable, timed out, returned `408`/`429`/`5xx`, or is circuit-open or saturated; any other `4xx` (`400`, `401`, `403`, `404`, `422`, …) is returned to the caller instead. Named modes — `single`, `conditional`, `content-based` — stay inside what was named: `single` reports its one target's failure, and a `conditional` or `content-based` rule walks its `target_keys` chain on the same failover-safe failures without ever reaching a target it did not name. Every mode skips a target whose circuit breaker is open, or that is parked after a `429`, among the candidates it offers; if every target for a model is open, the request gets `503`. `targets[].retry` is honored under every mode, so set `attempts: 1` to keep single-attempt behavior even in a pool.

</div>
</details>
Expand Down
Loading
Loading