feat(catalog): add agent-routing metadata tiers/aliases/benchmarks - #24
feat(catalog): add agent-routing metadata tiers/aliases/benchmarks#24gr3enarr0w wants to merge 1 commit into
Conversation
…erro-labs#23) Adds optional, catalog-owned metadata so coding agents and local routing clients can pick models without hard-coding private tier tables: - agent_routing: coding/reasoning/tool_use/latency tiers, local_suitability, recommended_roles — enum-validated (missing values always valid). - aliases: routing surface -> resolved model IDs (incl. Ferro Gateway). - benchmarks: coding (swe-bench|local|other) + local_runtime (GGUF/llama.cpp). Schema: - Entry gains *AgentRouting, Aliases, *Benchmarks (pointer-backed so omitempty drops the whole block when unset; JSON artifacts stay backward-compatible). - Extends deep-merge for agent_routing (per-field override), full-replacement for aliases/benchmarks (mirrors Pricing/Capabilities semantics). - Validate() rejects malformed enums but allows missing metadata. Docs: - README "Agent-routing metadata" section + docs/agent-routing.md with the full schema and a coding-agent routing client example. dist/ is intentionally not regenerated here — that is a release-time artifact produced by `make build` on merge. (On case-insensitive macOS, the two case-variant bge YAMLs collide on disk, so a local rebuild would emit a spurious together.json delta unrelated to this schema change.) Closes ferro-labs#23.
|
Preparing review... |
1 similar comment
|
Preparing review... |
|
Preparing PR description... |
|
Preparing review... |
1 similar comment
|
Preparing review... |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
Thanks for this — the implementation is genuinely clean. The pointer-backed blocks with I've raised a design question on #23 rather than here, because it's about where this data should live rather than how it's written: the tiers have no oracle behind them (nothing scrapes them, Some notes from reading the diff, mostly relevant either way:
None of these are blockers on their own. Let's sort out the #23 question first. |
Summary
Implements #23. Adds optional, catalog-owned metadata so coding agents and local routing clients can pick models without hard-coding private tier tables (the kind of logic SynapseRouter used to embed in a router-specific codebase).
Fields added to
Entry(all optional, pointer-backed soomitemptydrops the whole block when unset — JSON artifacts stay backward-compatible)agent_routingcoding_quality_tier,reasoning_quality_tier(frontier|strong|balanced|fast|experimental|unknown),tool_use_quality_tier(strong|balanced|weak|unknown),latency_tier(low|medium|high|unknown),local_suitability(excellent|good|poor|unknown),recommended_roles[]stringaliasesbenchmarkscoding(source swe-bench|local|other, score, updated_at) +local_runtime(quantization, backend, tokens_per_second, hardware)Behavior
agent_routingdeep-merges per-field (wrapper overrides individual tiers without restating the block);aliases/benchmarksuse full-replacement semantics (same rationale asPricing/Capabilities— YAML can't distinguish "unset" from "explicitly empty" for map/struct fields).TestEntry_AgentRoutingJSONRoundTrip).Docs
docs/agent-routing.mdwith the full schema, an extends-inheritance note, an example JSON artifact, and a TypeScript coding-agent routing-client example (acceptance: documentation includes examples for coding-agent routing clients).Testing
go test -race ./...— new suiteagent_routing_test.gocovers: full valid block passes, missing block (back-compat) passes, every malformed enum is rejected, extends merge inherits/overrides correctly, JSON round-trip surfaces and omits keys correctly.go vet ./...clean.go run ./cmd/ferrocat validatereports0 errorsagainst the live catalog.Non-goals enforced
Notes for reviewers
dist/is not regenerated in this PR. It's a release-time artifact produced bymake buildon merge (CI runs the same). On case-insensitive macOS the two case-variantbge-base-en-v1.5YAMLs (BAAI__/baai__) that already exist inproviders/together/models/collide on disk, so a localmake buildwould emit a spurioustogether.jsondelta unrelated to this schema change; I left dist untouched to keep the diff focused.TestRoundTripRealCatalogdiscrepancy (1 entry: original 2479 vs rebuilt 2478) exists on cleanorigin/maintoo and is not introduced here.Closes #23.