Skip to content

feat(catalog): add agent-routing metadata tiers/aliases/benchmarks - #24

Open
gr3enarr0w wants to merge 1 commit into
ferro-labs:mainfrom
gr3enarr0w:feat/agent-routing-metadata-23
Open

feat(catalog): add agent-routing metadata tiers/aliases/benchmarks#24
gr3enarr0w wants to merge 1 commit into
ferro-labs:mainfrom
gr3enarr0w:feat/agent-routing-metadata-23

Conversation

@gr3enarr0w

Copy link
Copy Markdown
Contributor

Summary

Implements #23. Adds optional, catalog-owned metadata so coding agents and local routing clients can pick models without hard-coding private tier tables (the kind of logic SynapseRouter used to embed in a router-specific codebase).

Fields added to Entry (all optional, pointer-backed so omitempty drops the whole block when unset — JSON artifacts stay backward-compatible)

Block Fields
agent_routing coding_quality_tier, reasoning_quality_tier (frontier|strong|balanced|fast|experimental|unknown), tool_use_quality_tier (strong|balanced|weak|unknown), latency_tier (low|medium|high|unknown), local_suitability (excellent|good|poor|unknown), recommended_roles []string
aliases routing surface → resolved model IDs (incl. Ferro Gateway)
benchmarks coding (source swe-bench|local|other, score, updated_at) + local_runtime (quantization, backend, tokens_per_second, hardware)

Behavior

  • Validate() rejects malformed enum values but allows missing metadata (acceptance criterion). Existing models without the block validate unchanged.
  • Extends: agent_routing deep-merges per-field (wrapper overrides individual tiers without restating the block); aliases / benchmarks use full-replacement semantics (same rationale as Pricing / Capabilities — YAML can't distinguish "unset" from "explicitly empty" for map/struct fields).
  • JSON artifacts surface the new top-level keys, omitted entirely when unset (verified by TestEntry_AgentRoutingJSONRoundTrip).

Docs

  • README gets an "Agent-routing metadata" subsection.
  • New docs/agent-routing.md with the full schema, an extends-inheritance note, an example JSON artifact, and a TypeScript coding-agent routing-client example (acceptance: documentation includes examples for coding-agent routing clients).

Testing

  • go test -race ./... — new suite agent_routing_test.go covers: full valid block passes, missing block (back-compat) passes, every malformed enum is rejected, extends merge inherits/overrides correctly, JSON round-trip surfaces and omits keys correctly.
  • go vet ./... clean.
  • go run ./cmd/ferrocat validate reports 0 errors against the live catalog.

Non-goals enforced

  • No Pi-specific routing policy in the catalog — only reusable metadata.
  • Local/private benchmark data is optional for public models.

Notes for reviewers

  • dist/ is not regenerated in this PR. It's a release-time artifact produced by make build on merge (CI runs the same). On case-insensitive macOS the two case-variant bge-base-en-v1.5 YAMLs (BAAI__ / baai__) that already exist in providers/together/models/ collide on disk, so a local make build would emit a spurious together.json delta unrelated to this schema change; I left dist untouched to keep the diff focused.
  • A pre-existing TestRoundTripRealCatalog discrepancy (1 entry: original 2479 vs rebuilt 2478) exists on clean origin/main too and is not introduced here.

Closes #23.

…erro-labs#23)

Adds optional, catalog-owned metadata so coding agents and local routing
clients can pick models without hard-coding private tier tables:

- agent_routing: coding/reasoning/tool_use/latency tiers, local_suitability,
  recommended_roles — enum-validated (missing values always valid).
- aliases: routing surface -> resolved model IDs (incl. Ferro Gateway).
- benchmarks: coding (swe-bench|local|other) + local_runtime (GGUF/llama.cpp).

Schema:
- Entry gains *AgentRouting, Aliases, *Benchmarks (pointer-backed so omitempty
  drops the whole block when unset; JSON artifacts stay backward-compatible).
- Extends deep-merge for agent_routing (per-field override), full-replacement
  for aliases/benchmarks (mirrors Pricing/Capabilities semantics).
- Validate() rejects malformed enums but allows missing metadata.

Docs:
- README "Agent-routing metadata" section + docs/agent-routing.md with the
  full schema and a coding-agent routing client example.

dist/ is intentionally not regenerated here — that is a release-time artifact
produced by `make build` on merge. (On case-insensitive macOS, the two
case-variant bge YAMLs collide on disk, so a local rebuild would emit a spurious
together.json delta unrelated to this schema change.)

Closes ferro-labs#23.
@gr3enarr0w
gr3enarr0w requested a review from MitulShah1 as a code owner June 29, 2026 00:21
@gr3enarr0w

Copy link
Copy Markdown
Contributor Author

Preparing review...

1 similar comment
@gr3enarr0w

Copy link
Copy Markdown
Contributor Author

Preparing review...

@gr3enarr0w

Copy link
Copy Markdown
Contributor Author

Preparing PR description...

@gr3enarr0w

Copy link
Copy Markdown
Contributor Author

Preparing review...

1 similar comment
@gr3enarr0w

Copy link
Copy Markdown
Contributor Author

Preparing review...

@gr3enarr0w

Copy link
Copy Markdown
Contributor Author

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis ✅

23 - Fully compliant

Compliant requirements:

  • Catalog schema supports optional agent-routing/tier metadata without requiring it on every model.
  • Catalog schema supports provider/client aliases, including Ferro Gateway aliases.
  • Catalog schema supports local benchmark artifacts from GGUF/llama.cpp testing.
  • JSON artifacts expose the new top-level keys for downstream clients.
  • Validation rejects malformed enum values but allows missing metadata.
  • Documentation includes examples for coding-agent routing clients.
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected

@MitulShah1

Copy link
Copy Markdown
Contributor

Thanks for this — the implementation is genuinely clean. The pointer-backed blocks with omitempty, the per-field extends merge, and the enum table tests are all the right calls.

I've raised a design question on #23 rather than here, because it's about where this data should live rather than how it's written: the tiers have no oracle behind them (nothing scrapes them, freshness can't detect drift), which sits awkwardly next to the rest of the catalog. I'm proposing we split the benchmarks out (those are citable facts, happy to take them) and move the tiers/roles/aliases into a separate routing overlay artifact. Would rather settle that before you spend more time on the schema — please have a look and push back if you disagree.

Some notes from reading the diff, mostly relevant either way:

  • Score and TokensPerSecond are float64 with omitempty, so a legitimate 0.0 disappears from the JSON. NullFloat64 exists in this repo for exactly that reason.
  • gofmt isn't clean on types.go, validate.go, and the new test file (make fmt fixes it). main is currently clean.
  • The branch has drifted from main and now conflicts — the README edit still says "193 wrapper models" where main says 269.
  • CI never ran on this, so nothing here has actually been gated.
  • The pre-existing TestRoundTripRealCatalog discrepancy you mention doesn't reproduce for me on current main — the test passes. Might have been real against the older base; worth rechecking after a rebase.
  • local_suitability: excellent on gpt-5-pro in the README example doesn't quite make sense for a closed API model.

None of these are blockers on their own. Let's sort out the #23 question first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add agent-routing metadata for coding tiers, local benchmarks, and aliases

2 participants