[v5] RFC: constellation — what a hub needs to coordinate several spokes as one fleet

[hanthor]

Summary

A single hive serving a diverse org makes its agents compete over unrelated
work. Splitting into several subject-scoped hives fixes that, but the moment
you do, you discover the hub is a directory, not a coordinator — and
several things that ought to be fleet-wide are per-hive by accident rather than
by design.

This RFC reports what actually broke splitting one 43-repo hive into two, and
proposes what a constellation — several spokes under one hub, acting as one
system — would need.

We are running this now: a self-hosted hub, two spokes, shared credentials.
Everything below is measured, not speculative.

What we built

                    hub.tunaos.org        (HIVE_MODE=hub, registry)
                          |
        +-----------------+-----------------+
        |                                   |
  hive.tunaos.org                     reef.tunaos.org
  25 repos · ACMM L6                  18 repos · ACMM L5
  OS, images, installers,             desktop apps, app
  packaging                           distribution, docs
        |                                   |
        +--------- shared -------------------+
          GitHub App installation
          agent OAuth credentials

Spokes register themselves by heartbeat once they have HIVE_HUB_URL and the
hub's HIVE_HUB_SECRET. That part works exactly as documented.

What the hub does NOT coordinate

Each of these is fleet-wide in reality but per-hive in the implementation.

1. Provider quota

Every spoke probes and consumes provider quota independently, with no shared
view. Two spokes on one Anthropic subscription each believe they have the whole
weekly cap; the first to notice exhaustion is the one that hits it.

This is the single biggest constraint on splitting. A split creates zero
additional quota — it divides the same pool among more agents. Ours:
DeepSeek dry, OpenAI capped for four days, Anthropic near its weekly cap,
Google carrying the fleet and hitting its own 5-hour window.

Constellation need: headroom as hub state, so a spoke can ask "what is left
for the fleet" rather than "what is left as far as I can see".

2. GitHub App rate limit

Both spokes use one App installation and therefore share its 7100 req/hr.
Bringing up the second spoke exhausted it during initial enumeration:

403 API rate limit exceeded for installation ID 137498420 [rate reset in 19m22s]
... all 18 repos failed to enumerate

Nothing coordinates this. Each spoke polls its own repos on its own schedule
and discovers the shared ceiling by hitting it.

Constellation need: either a documented per-spoke budget, or hub-level
awareness so spokes back off cooperatively instead of racing.

3. Agent credentials

Each hive pod has its own /data/home, so by default every spoke needs its
own interactive claude /login — and those sessions rotate refresh tokens
against each other. We saw exactly that failure earlier: a copied credential
was zeroed when the other session refreshed, and Claude Code silently fell back
to metered API billing.

We solved it by mounting one credential directory into both spokes, which works
because a single hive already shares ~/.claude across its 11 agents by
symlink. Extending that across hives is the same model.

But it needs both spokes on one node (RWO binds to a node), and PVCs are
namespaced — we needed a second PV over the same hostPath to cross the
namespace boundary. That is a workaround, not a design.

Constellation need: a first-class notion of fleet-shared credentials.

4. Repo ownership

Nothing prevents two spokes managing the same repo, which would put two fleets
on the same backlog opening competing PRs — the exact thing a split exists to
prevent.

Our org watcher nearly caused this: it read one spoke's repo list, could not
read the other's (no owner session), concluded 18 repos were unmanaged, and was
one step from adding them to the first hive.

Constellation need: the hub should know which spoke owns a repo, and refuse
or warn on overlap. The registry already receives each spoke's repo list on
every heartbeat — the data is there, unused.

5. Routing new work

When a repo appears in the org, something must decide which spoke gets it. We
do this by asking an AI with each hive's charter in the prompt, falling back
across backends when one is rate-limited, and leaving a repo unassigned
rather than guessing — a repo in the wrong hive is worse than one nobody has
picked up.

That is spoke-side today and duplicated per operator. It is a natural hub
responsibility: the hub already knows every spoke and its charter.

Proposal

Sequenced smallest-first; each step is useful alone.

1. Overlap detection (cheap, high value). The hub already receives repo
lists via heartbeat. Flag a repo claimed by more than one spoke in
/api/registry and the hub UI. Pure read-side; no new protocol.

2. Spoke charters in the registry. Let a spoke advertise a one-line scope
alongside its repo list. Makes the registry self-describing and gives any
router — AI or human — the data it needs.

3. Fleet headroom. Let spokes report provider headroom on the heartbeat
they already send, and expose the aggregate. Read-only first; a spoke can
decide what to do with the number.

4. Shared-credential guidance. Document the one-directory-many-readers
model, with the node/namespace constraints spelled out. Ideally a supported
config field rather than a hostPath trick.

5. Hub-side routing (optional). Only once 1–3 exist. POST /api/route
taking a repo and returning a spoke, so every operator does not reimplement it.

Deliberately not proposed: the hub scheduling work onto spokes, or holding
credentials. It is a directory; making it a control plane changes its failure
model from "registry is stale" to "fleet is down".

Open questions

1. Is per-spoke GitHub App installation the intended answer to the rate-limit
   ceiling? It solves it cleanly but multiplies App management.
2. Should overlap be an error or a warning? Legitimate cases may exist (a docs
   repo two subject areas both touch).
3. Does hub-side routing belong here at all, or in a separate tool that reads
   the registry?
4. Is there appetite for spokes reporting quota to the hub, given it is
   operator-sensitive data?

Environment

ghcr.io/kubestellar/hive-hub:stable + two hive:stable spokes on a 2-node
Talos cluster; ACMM L6 and L5; one GitHub App installation; shared Claude Max
and Antigravity credentials. Related: #5632, #5633, #5661, #5690.

[clubanderson]

Dispatching this RFC against v5 tip 67430b2222f7552a6fc93372ef49f21887f3e9ee.

Assessment of current v5:

• The hub already has a heartbeat-backed registry. src/pkg/hub/heartbeat.go receives HeartbeatPayload with Org, Repos, PrimaryRepo, Agents, Governor, health, ProviderLimitReason, component reach, GitHub App identity, dashboard URL, and more. HeartbeatProjectConfig already pushes hub-side project config back to spokes on the heartbeat response.
• src/pkg/hub/server.go's RegistryEntry persists/display fields such as Org, Repos, PrimaryRepo, ClusterID, Owner, DashboardURL, GitHub App state, agent summaries, provider-limit reason, and duplicate reporter/status-flip signals. There is no charter/scope field and no explicit repo-overlap index.
• Fleet/registry UI and operations already exist hub-side (/fleet, registry pages, bulk operations in src/pkg/hub/saas_bulk.go). Bulk operations intentionally use heartbeat delivery for pull-only spokes, which fits this RFC's directory-not-control-plane boundary.
• GitHub setup routing already finds candidate hives by org/repos (src/pkg/hub/gh_setup.go, src/pkg/hub/webhook.go), but it does not enforce single ownership across the whole registry or route new repos by charter.
• Provider quota data is currently per-spoke/provider-limit signaling (ProviderLimitReason, ProviderLimitRebuffs) plus the spoke-side rotation/headroom work in src/pkg/rotation; it is not hub-aggregated fleet state.
• Shared credentials are not a hub abstraction today. Spoke data config has shared in-pod homes such as /data/home/Claude sessions in src/pkg/config/config.go, but cross-spoke credential placement remains operator/deployment guidance.

Phased v5 sketch:

1. Overlap detection first: build a read-side registry index of normalized (github_host, org, repo) → hives from RegistryEntry.Repos/PrimaryRepo; surface overlaps in /api/registry and hub UI as warnings. No heartbeat protocol change required, and no scheduling/control-plane semantics.
2. Add spoke charters: extend HeartbeatPayload and RegistryEntry with an optional short charter/scope string, sourced from config and sanitized like other spoke-reported display fields. Show it in the fleet/registry UI and include it in registry API responses.
3. Fleet headroom reporting: once [v5] RFC: backend capacity, model inventory, and placement #5698's normalized headroom shape exists, add an optional heartbeat field for provider headroom summaries and store it on RegistryEntry. Start read-only: aggregate by provider/account key where known, mark unknown explicitly, and do not let the hub drive rotations.
4. GitHub App rate-limit visibility: add a spoke heartbeat field for installation rate-limit remaining/reset when available, or a hub-side registry aggregate keyed by App installation/App ID. Render budget pressure and document the per-spoke-installation escape hatch.
5. Shared credential guidance/config: document the supported one-directory-many-agents model and the Kubernetes/RWO/namespace constraints. If we add config, keep it as a spoke/deployment path knob; do not make the hub hold credential material.
6. Optional route API: after overlap+charter+headroom exist, add a hub read/recommend endpoint such as POST /api/route that takes host/org/repo and returns ranked spokes plus reasons. Leave final assignment to an operator/spoke until the failure model is intentionally changed.

Open questions for @hanthor:

• For overlap detection, should duplicates block assignment/provisioning, or start as a visible warning with an allowlist for intentionally shared repos?
• What should identify shared provider quota when aggregating headroom: provider only, account/email hash, GitHub App installation, operator-supplied pool ID, or some combination?
• Should charters live in project config, hub-side SaaSHive metadata, or both with provenance? Who should be allowed to edit them?
• For GitHub App rate limits, is the preferred long-term answer separate installations per spoke, or cooperative budgeting for a shared installation?
• Should the router ever mutate spoke config, or remain recommendation-only as this RFC suggests?

[hanthor]

Thanks — the phasing matches what we'd do, and starting overlap detection read-side off data the heartbeat already carries is the right first cut.

Answers, in order, all from running this rather than from preference:

1. Overlap: warn + allowlist, not block.

Intentional sharing is real and we do it deliberately — tuna-os/hive is in both our spokes on purpose, so each can file feedback upstream from its own context. A hard block would have to be overridden on day one.

The distinction that matters isn't shared-vs-not, it's who chose. An operator who lists a repo in two spokes has made a decision; an auto-router that assigns an already-claimed repo has not. So: warn on operator-declared overlap with an allowlist to silence it, and refuse on automatic assignment to an already-claimed repo. That also keeps step 1 honest as a read-side feature — the refusal lives in step 6's router, not in the index.

2. Shared quota identity: operator-supplied pool ID, defaulted from the credential source.

This is the one with no good automatic answer, and I'd rather say so than have the hub guess.

Provider alone is wrong: two spokes on two different Anthropic accounts are not one pool, and aggregating them would under-report headroom badly. Account identity isn't available either — api.anthropic.com/api/oauth/usage returns limits with percent / resets_at and no account identifier, so there is nothing to key on from the probe.

What is reliable is that spokes sharing a credential are definitionally sharing a pool. That's exactly why ours share: one mounted credential directory, both spokes reading it. So the useful default is to derive the pool key from the credential source (mount path / secret ref), with an explicit operator-supplied pool_id override for cases the deployment can't see. Explicit ID wins; credential-derived default covers the common case; provider-only as the last resort, marked low-confidence.

One correction to step 3, which we found today and which changes its shape: headroom is not a scalar. Anthropic returns several concurrent limits on independent clocks:

percent 37   resets 19:30   (5h session)
percent 67   resets 23:00   (weekly)
percent 100  resets 23:00   scope.model="Fable"   (model-class cap)

Three consequences for a headroom heartbeat field:

• Collapsing to a max loses the binding constraint. 63% headroom over 2.6h and 33% over 6.1h are very different allowances; which one binds depends on the burn rate, not on which percent is larger.
• Model-scoped limits must be excluded from provider headroom. We took the max across all limits and a weekly_scoped{model} at 100% read as total exhaustion — it parked an 11-agent fleet while Sonnet was answering normally throughout.
• So the field wants to be a list of {percent, resets_at, scope}, not a number. The consumer takes the min over headroom_i / time_to_reset_i and ignores scoped entries when asking "can this provider serve at all".

Happy to send that shape as a concrete patch against whatever you land for step 3.

3. Charters: spoke project config, hub-side override with provenance.

The spoke operator knows the scope, and keeping it next to repos/primary_repo means it can't drift from the repo list it describes. But a fleet operator retiring or re-scoping a spoke needs to change it without a spoke redeploy, so a hub-side override recorded with provenance (source: spoke|hub) covers that. Edit rights: hive owner for the spoke value, hub operator for the override.

4. GitHub App rate limits: separate installations as the recommendation, plus one cheap cooperative fix.

Our data point is a bringup failure, not steady-state pressure. Standing up the second spoke against the shared installation:

403 API rate limit exceeded for installation ID 137498420 [rate reset in 19m22s]
... all 18 repos failed to enumerate

Steady state on the same installation is fine. What broke was initial enumeration — a new spoke walking its whole repo list at once, against a budget already carrying the first spoke's normal load.

So I'd separate the two answers. Long-term isolation: separate installations, because it needs no coordination protocol and no shared-state correctness argument. But that alone wouldn't have prevented this, and plenty of orgs will want one App. The cheap fix that would have: stagger and jitter first-enumeration, and surface installation_rate_limit_remaining / reset on the heartbeat so the hub can show budget pressure before a spoke discovers the ceiling by hitting it. Cooperative budgeting beyond that (spokes negotiating shares) feels like more mechanism than the problem justifies.

5. Router stays recommendation-only.

Two reasons. A wrong assignment is worse than no assignment — that's why ours leaves a repo unassigned rather than guessing, and a hub that writes has to be right without the spoke's context. And the spoke's repo list is its own config; a hub write creates a split-brain the next spoke deploy silently resolves the wrong way. Advisory keeps the failure mode "nobody picked it up", which is visible and cheap, instead of "two fleets on one backlog", which isn't.

If the hub ever should mutate, I'd want it gated behind an explicit per-spoke opt-in, so it's a spoke delegating rather than a hub asserting — but I wouldn't build that until 1–3 have been in use for a while.

One note on ordering: I'd put charters (your 2) before overlap warnings (your 1) if they're close in cost, only because a charter makes an overlap warning actionable — "both spokes claim this repo" plus two one-line scopes tells you which one should keep it, where the bare warning doesn't.

[clubanderson]

Progress note: phase 1 from the earlier dispatch is now delivered by #5705 and merged to v5. The hub derives repo-ownership overlaps from the registry read path and reports them in /api/registry plus warning fleet alerts, without changing heartbeat protocol or blocking assignment. No re-dispatch here; keeping the remaining constellation/RFC items open.

[clubanderson]

Closed by merged v5 PR #5796, which lands the accepted constellation RFC and records #5705 as the delivered phase-1 overlap detection.

[clubanderson]

Roadmap anchor follow-up: the constellation RFC is now reflected in the v4 canonical roadmap and the public Now/Next/Later roadmap, with the v5 port merged too.

• Root roadmap v5 workstreams: https://github.com/hivecommons/hive/blob/v4/ROADMAP.md#v5--next-generation
• Public roadmap Now row: https://github.com/hivecommons/hive/blob/v4/src/docs/roadmap.md#now
• v5 public roadmap port: https://github.com/hivecommons/hive/blob/v5/src/docs/roadmap.md#now
• Production adopter entry for tunaos.org: https://github.com/hivecommons/hive/blob/v4/ADOPTERS.md#adopters-list

Merged in #5830 and #5831; #5773 is closed.