Summary
The current Team CRD bundles two distinct concerns into a single resource:
- Organizational relationships — who is in the team, who is the leader, who is admin, peer-mention policy.
- Per-member runtime definition — model, identity, soul, package, mcpServers, env, accessEntries, etc. for every leader/worker.
This bundling has produced concrete pain points: triplicated API types, an over-burdened TeamReconciler, no independent K8s identity for team members, and tight coupling between organizational changes and runtime state.
This issue proposes splitting the two concerns:
Team CR keeps only organizational/policy fields (workerMembers[], admin, humanMembers, accessPolicy, channelPolicy).
- Every team member becomes a regular
Worker CR, owned by WorkerReconciler exclusively.
TeamReconciler becomes a thin coordinator that only manages Matrix Room membership, MinIO AGENTS.md coordination injection, and status aggregation. It never touches Pods, ServiceAccounts, configs, or routes.
- Crucially,
Worker CRs carry no team annotation. Team membership is expressed only by Team.spec.workerMembers[].name, queried via field indexer.
We have a working PoC of this design (controller + dual-format API + automatic migrator) running in our internal integration environment. End-to-end validation passed across legacy-format, decoupled-format, and mixed scenarios with no Pod rebuilds and no Matrix Room recreation. We would like to upstream this to HiClaw and are opening this issue first to gather feedback on the architecture before sending PRs.
1. Current state (Before)
1.1 The "all-in-one" Team CR
type TeamSpec struct {
Description string `json:"description,omitempty"`
TeamName string `json:"teamName,omitempty"`
Admin *TeamAdminSpec `json:"admin,omitempty"`
HumanMembers []TeamMemberSpec `json:"humanMembers,omitempty"`
Leader LeaderSpec `json:"leader"` // 15 runtime fields inlined
Workers []TeamWorkerSpec `json:"workers,omitempty"` // 18 runtime fields inlined per member
PeerMentions *bool `json:"peerMentions,omitempty"`
ChannelPolicy *ChannelPolicySpec `json:"channelPolicy,omitempty"`
}
Team members do not have their own Worker CR today. TeamReconciler reads runtime definitions directly from Team.spec.leader / Team.spec.workers[] and converges Matrix users, configs, Pods, exposed ports, etc. This means:
- "Who belongs to the team" and "how that member runs" are the same Kubernetes object.
- A standalone
Worker and a team-member Worker are produced by completely different code paths (WorkerReconciler vs TeamReconciler calling shared phases).
1.2 Concrete problems
Problem 1 — Three near-duplicate API types
| struct |
fields |
purpose |
WorkerSpec |
18 |
standalone Worker runtime |
TeamWorkerSpec |
18 |
team worker runtime (≈1:1 duplicate) |
LeaderSpec |
15 |
team leader runtime (subset of WorkerSpec) |
Adding any new runtime field (e.g. RemoteSkills) requires changes in all three structs plus the teamWorkerSpecToWorkerSpec() conversion, plus REST request types. Missing one location silently breaks team workers. LeaderSpec is also artificially restricted (no Image, no Skills[] list, no Expose) — those gaps exist purely because keeping three structs in sync is expensive.
There is also a hashMemberSourceSpec invariant: every new field must use omitempty with the right zero-value semantics, otherwise upgrading the controller triggers cluster-wide container rebuilds. The code carries a 60-line comment specifically warning about this load-bearing constraint.
Problem 2 — TeamReconciler is too heavy
team_controller.go is over 1000 lines. A single reconcile chains six steps:
reconcileTeamNormal:
1. Provision Team Room / Leader DM / Shared Storage (team infra)
2. Write inline configs (Identity/Soul/Agents to MinIO) (member runtime)
3. Cleanup stale members (member runtime)
3.5 Inject coordination context (member runtime)
4. For each member -> shared phases: (member runtime, the bulk)
Matrix user / SA / Config / Container / Expose
5. Update legacy registry (member metadata)
6. Aggregate status (status)
Steps 2-5 are ~80% of the code and are conceptually about per-member runtime, not team organization. Consequences:
- Changing one member's
model triggers a Team-level reconcile.
- A single backend failure tips the whole Team into
Degraded.
- Pod watch is broad: every Pod with a
hiclaw.io/team label triggers TeamReconciler.
- Debugging a worker startup means reading Team reconcile logs.
Problem 3 — No independent K8s identity for team members
Team members exist only as elements of Team.spec.workers[]. There is no Worker CR for them. As a result:
kubectl get worker <team-member> does not work.
GET /workers/<team-member> requires server-side synthesis from the Team CR.
- The auth enricher needs a field indexer to reverse-look-up Team CRs because there is no Worker CR to read.
accessEntries are scattered at different nesting depths (spec.leader.accessEntries, spec.workers[i].accessEntries).
Problem 4 — Organizational changes interfere with runtime state
Adding/removing a member rewrites the same CR that carries every other member's runtime spec. Concretely:
- Adding a new member triggers communication-policy and hash recomputation for every existing member.
- Changing
ChannelPolicy ripples through every member container.
- Deleting the Team is effectively "delete all members' runtime."
- Editing one member's model triggers a Team-level reconcile.
Problem 5 — Dual code paths for runtime work
Every feature touching worker runtime (skills, exposed ports, lifecycle, etc.) has to be wired through two different controllers. Easy to drift, easy to forget one side.
1.3 Summary table
| Domain |
Issue |
Root cause |
Impact |
| API design |
Three duplicate types |
Team CR embeds runtime defs |
High evolution cost; Leader feature-gapped |
| Controller arch |
TeamReconciler over-burdened |
Org + runtime in one loop |
Whole-team coupling, hard debugging, no per-member retry |
| Resource model |
No member-level CR |
Inline design choice |
Standard kubectl unusable, API hacks, indexer-based auth |
| Concerns |
Org change disturbs runtime |
Single CR, two responsibilities |
Member changes are destructive |
| Engineering |
Two code paths |
Two controller entry points |
Low velocity, drift risk |
2. Proposed design (After)
2.1 Core principles
- Workers are team-agnostic. A
Worker CR has zero team-related annotations or labels. WorkerReconciler never reads a Team CR.
- Team is organization + policy only.
Team.spec lists references to existing Worker CRs and team-level policies — nothing about runtime.
- Every agent has a Worker CR. Standalone or team member, runtime is owned by
WorkerReconciler alone.
- Team semantics are delivered out-of-band.
TeamReconciler invites/kicks members in the Matrix Team Room and writes a coordination block into each member's MinIO AGENTS.md. It does not patch the Worker CR or recreate the Pod.
- Pod ownerReference points to the Worker CR, not the Team.
- Permissions are merged at issuance time — Worker.spec.accessEntries union Team.spec.accessPolicy.defaultEntries.
- Migration is gradual and additive — old Team CRs remain functional during a compatibility window; a controller-internal migrator advances them.
2.2 Architecture overview
+---------------------------------------------------------------------+
| User / Control-plane API |
+---------------------------------------------------------------------+
| POST /workers -> Worker CR (runtime definition) |
| POST /teams -> Team CR (workerMembers[] + accessPolicy + ...)|
+----------+---------------------------------------+------------------+
| |
+------v------+ +-----v------+
| Worker CR | <- TeamReconciler | Team CR |
| (runtime) | reads Worker.status | (org+policy|
| no team ann | only | ) |
+------+------+ +-----+------+
| |
+------v---------+ +-----v-----------+
| WorkerRecon | | TeamReconciler |
| team-agnostic | | out-of-band |
| Pod / config / | | delivery only: |
| matrix user / | | - Matrix |
| expose | | invite/kick |
+----------------+ | - MinIO |
| AGENTS.md |
| - status agg |
+-----------------+
2.3 New CRD shape
TeamSpec (after):
type TeamSpec struct {
Description string `json:"description,omitempty"`
TeamName string `json:"teamName,omitempty"`
Admin *TeamAdminSpec `json:"admin,omitempty"`
HumanMembers []TeamMemberSpec `json:"humanMembers,omitempty"`
// Reference existing Worker CRs (replaces inlined runtime definitions).
WorkerMembers []TeamWorkerRef `json:"workerMembers,omitempty"`
// Team-level access policy (replaces hard-coded DefaultEntriesForTeamMember).
AccessPolicy *TeamAccessPolicySpec `json:"accessPolicy,omitempty"`
PeerMentions *bool `json:"peerMentions,omitempty"`
ChannelPolicy *ChannelPolicySpec `json:"channelPolicy,omitempty"`
// Deprecated -- kept during the compatibility window. Ignored when WorkerMembers is set.
Leader LeaderSpec `json:"leader,omitempty"`
Workers []TeamWorkerSpec `json:"workers,omitempty"`
}
type TeamWorkerRef struct {
Name string `json:"name"` // metadata.name of the referenced Worker CR
Role string `json:"role,omitempty"` // "team_leader" | "worker" (default "worker")
}
type TeamAccessPolicySpec struct {
// Merged with Worker.spec.accessEntries at credential issuance time.
DefaultEntries []AccessEntry `json:"defaultEntries,omitempty"`
}
WorkerSpec / WorkerStatus: unchanged. Worker.status.matrixUserID and Worker.status.roomID already exist and are read by TeamReconciler.
Worker metadata: contains no team-related annotation:
hiclaw.io/team (removed)
hiclaw.io/role (removed)
hiclaw.io/team-leader (removed)
hiclaw.io/team-admin-id (removed)
Membership is expressed only via Team.spec.workerMembers[] and discovered through a spec.workerMembers[*].name field indexer.
Validation rules:
workerMembers[].name must be unique within a Team.
- Exactly one
role: team_leader is required.
- Empty
role defaults to worker.
- A missing referenced Worker CR makes the Team
Degraded.
- Initial constraint: a Worker may belong to at most one Team. This is enforced via the field indexer (multiple hits -> error). Multi-Team is not blocked by design and can be relaxed later.
2.4 Reconciler responsibilities
TeamReconciler (after) does only:
- Validate membership (refs exist, exactly one leader, no duplicates).
- Provision team infrastructure (Team Room, Leader DM, shared storage).
- Manage Matrix room membership (read
Worker.status.matrixUserID, invite/kick).
- Inject coordination block into every member's MinIO
AGENTS.md (different content for leader vs worker — same as today).
- Update legacy registries.
- Aggregate status from
Worker.status.
It no longer: patches Worker CRs, creates Matrix users for workers, deploys configs, creates/recreates Pods, manages exposed routes, or manages ServiceAccounts.
WorkerReconciler (after) is unified for all workers and remains team-agnostic:
- Read
Worker.spec.
- Build a standalone
MemberContext (no TeamName, no peer list, no Team channel policy).
- Run shared phases (Infra, ServiceAccount, Config, Container, Expose).
- Write
Worker.status.
2.5 Coordination delivery (no Pod rebuild)
Instead of patching CRs to communicate team membership to the agent process, we use two existing channels:
- Matrix Room invite/kick — agents auto-accept invites via Matrix sync; getting kicked is observable as a room-leave event.
- MinIO
AGENTS.md injection — TeamReconciler writes a <!-- hiclaw-team-context-start --> ... <!-- hiclaw-team-context-end --> block to each member's agents/<name>/AGENTS.md. The agent picks this up via mc mirror (seconds-level) and hot-reloads.
Block content stays role-differentiated, identical in spirit to today:
- Leader gets the full picture: team room, leader DM, worker list, heartbeat/idle timeouts, behavioral rules.
- Worker gets only its coordinator + admin + behavioral rules — no peer information.
Adding/removing a member therefore:
- Patches the Team CR (
workerMembers += {...}).
TeamReconciler invites the new member's Matrix user to the Team Room.
TeamReconciler writes/updates MinIO blocks for affected members.
No Worker reconcile, no Pod restart.
2.6 Permission resolution
func (r *Resolver) ResolveForCaller(ctx context.Context, caller *CallerIdentity) (...) {
workerEntries := r.readWorkerEntries(ctx, caller.Username) // always
if caller.Team != "" {
teamEntries := r.readTeamDefaultEntries(ctx, caller.Team)
if teamEntries == nil {
teamEntries = DefaultEntriesForTeamMember(caller.Username, caller.Team) // fallback
}
return union(workerEntries, teamEntries), nil
}
return workerEntries, nil
}
- Team
accessPolicy.defaultEntries carries shared paths (e.g. teams/${self.team}/*, shared/*).
- Worker
spec.accessEntries carries personal paths (e.g. agents/${self.name}/*).
- Union, no conflict resolution needed (paths are namespaced).
DefaultEntriesForTeamMember() becomes a fallback used only when a Team has no explicit accessPolicy.
2.7 REST API surface
Workers:
| Endpoint |
Before |
After |
POST /workers |
rejects names that match team members |
always allowed; team members are also Workers |
GET /workers/{name} |
synthesizes from Team CR for team members |
reads Worker CR directly |
GET /workers |
merges standalone + synthesized |
lists Worker CRs directly |
PUT /workers/{name} |
rejects team members |
allowed |
DELETE /workers/{name} |
rejects team members |
allowed (also drops the Team reference) |
Teams:
| Endpoint |
Before |
After |
POST /teams |
full inline runtime |
only org + policy fields |
PUT /teams/{name} |
edits everything |
edits only org + policy |
DELETE /teams/{name} |
cascades into all members |
tears down team infra; does not delete Workers |
2.8 Dual-format POST /teams during the migration window
To keep existing clients working, the create-team handler accepts both shapes:
- Legacy:
{leader: {...}, workers: [...]} -> converted into Worker CRs + workerMembers[] automatically.
- Decoupled:
{members: [{name, role, ...}, ...]} -> direct path.
Sending both at the same time is a 400 error. Validation includes "exactly one team_leader" and "no duplicate worker names".
3. Automatic migration (in-controller)
Existing Team CRs are migrated by the controller itself — no external CLI, no Pod rebuilds, no Matrix Room recreation.
3.1 State machine
NotStarted -> WorkerCRsCreated -> StatusSeeded -> PodsReparented
|
v
Migrated <- TeamSpecPatched <- CoordinationInjected
Each transition writes team.metadata.annotations["hiclaw.io/migration-phase"] so a crashed/restarted controller can resume.
| Transition |
Action |
NotStarted -> WorkerCRsCreated |
Create Worker CRs for the leader and each worker; add finalizer hiclaw.io/migration-in-flight on Team |
WorkerCRsCreated -> StatusSeeded |
Patch Worker.status (matrixUserID, roomID) so existing Matrix users / rooms stay bound to the new CR |
StatusSeeded -> PodsReparented |
JSON-Patch each member Pod's metadata.ownerReferences to point to its Worker CR |
PodsReparented -> CoordinationInjected |
Matrix invite all members + write MinIO coordination blocks |
CoordinationInjected -> TeamSpecPatched |
Write spec.workerMembers[] and spec.accessPolicy; old leader/workers fields kept |
TeamSpecPatched -> Migrated |
Remove finalizer; mark migrated |
3.2 Safety gates
- Cluster flag
--enable-auto-team-migration=true (default on).
- Per-Team annotation
hiclaw.io/auto-migrate=disabled to opt out, or =rollback to reverse.
- Health pre-check: Team
phase=Active and every member has non-empty MatrixUserID and RoomID.
- Token bucket on starting migrations (1 QPS, concurrency 1 by default). Once a Team enters the state machine it is not rate-limited — half-migrated states are not allowed to linger.
- Leader-elected: only the leader controller runs migrator steps.
3.3 "Move-then-write" atomicity
The new CRD schema drops legacy TeamMemberStatus fields (Observed, SpecHash, RoomID, ExposedPorts, RuntimeName) outright. We rely on K8s pruning semantics: pruning happens on the next mutating write, not on read. The migrator therefore enforces this order per Team:
GET Team CR (reads legacy Status.Members while it still exists in etcd).
- Create Worker CRs and seed
Worker.status from the legacy values.
- Reparent Pods.
- Inject coordination.
- Last step:
UPDATE the Team CR with the new shape — this is when pruning fires.
A crash before step 5 leaves the Team CR untouched and the Worker CRs idempotent; the migrator simply resumes.
3.4 Pod ownerReference reparent
Today's team-member Pods have ownerReferences[*].controller pointing at the Team CR, so kubectl delete team cascades into Pods. Post-migration the owner must be the Worker CR; otherwise:
- Deleting the Team would still nuke worker Pods.
- Deleting the Worker would leave Pods orphaned.
The reparent uses a single JSON Patch and is guarded by the hiclaw.io/migration-in-flight finalizer to prevent a delete during the window.
3.5 Observability
- Events:
Normal MigrationPhaseAdvanced / Warning MigrationPhaseFailed per transition.
- Metrics:
hiclaw_team_migration_phase{team,phase} gauge, hiclaw_team_migration_failures_total counter.
kubectl get team gains a MIGRATION printer column.
4. Acceptance criteria
A reasonable definition of "done" for the upstream change:
- No Worker CR carries
hiclaw.io/team, hiclaw.io/role, or hiclaw.io/team-leader annotations.
- Adding/removing a Team member produces zero
WorkerReconciler reconcile events for affected workers.
- Pod restart count is unchanged across membership changes.
- Coordination context propagates within seconds (mc mirror cadence) after a Team change.
- Issued STS tokens contain the union of Worker
spec.accessEntries and Team accessPolicy.defaultEntries.
- Removing a Worker from a Team leaves the Worker CR + Pod intact and switches the agent to standalone mode automatically.
- Deleting a Team CR cascades into Team infrastructure only; no Worker Pod is deleted.
- A pre-existing Team CR (no
workerMembers) keeps working via the legacy reconcile path.
- Auto-migration drives every legacy Team to
Migrated with no Pod or Matrix Room recreation.
- Post-migration, every member Pod's
ownerReference uniquely points at its Worker CR.
hiclaw.io/auto-migrate=rollback reverses to the legacy shape.
- Killing the controller mid-migration is safe; the next leader resumes from the last persisted phase.
5. Risks and mitigations
| Risk |
Mitigation |
| MinIO write fails -> member misses team context |
TeamReconciler retries; surface in Team.status.message; agent degrades to standalone |
mc mirror lag delays team-context delivery |
Mirror cadence is seconds; agents also use Matrix room-join events as a secondary signal |
| Field indexer cost at very large Team counts |
Backed by informer cache, O(1) lookup; revisit at 1000+ Teams |
| Worker referenced by multiple Teams (not initially supported) |
Indexer returns multiple hits -> enricher errors; admission webhook can also reject |
Old agents that don't hot-reload AGENTS.md |
Restart fallback works; document minimum agent version |
| Users assume "delete Team" stops workers |
Documented + UI hint; semantics: Workers continue independently |
Worker deleted while still referenced by Team.workerMembers |
Team flips to Degraded with a clear message |
| Pod reparent fails |
Atomic JSON Patch + exponential backoff + finalizer blocks Team deletion |
| State-machine failure loop |
Backoff + per-Team disable annotation + alerting metrics |
| Annotation patch failure on Worker |
Not applicable — this design never patches Worker annotations |
6. Suggested rollout order
Stage 0 -> Stage 1 -> Stage 2 -> Stage 3 -> Stage 4 -> Stage 5 -> Stage 6
Agent CRD WorkerReconciler TeamReconciler REST + Auto- Drop
hot-reload extension becomes new path Auth/Access migration legacy
prereq (no behavior team-agnostic path
change)
- Stage 0: Confirm the agent runtime can hot-reload
AGENTS.md coordination blocks without restart.
- Stage 1: Add
WorkerMembers and AccessPolicy to TeamSpec (deprecate Leader/Workers but keep them). No behavior change.
- Stage 2: Strip team-annotation reads from
WorkerReconciler. MemberContext is always standalone. Standalone Workers are unaffected.
- Stage 3:
TeamReconciler takes the new path when spec.WorkerMembers != nil: validate -> infra -> invite -> MinIO inject -> status. No shared phases called.
- Stage 4: Dual-format REST handler; CREnricher uses field indexer only; AccessResolver merges entries.
- Stage 5: Migrator state machine with the
CoordinationInjected phase; safety gates and rollback.
- Stage 6: Remove the legacy
buildDesiredMembers/reconcileMember path and (in a major version) drop LeaderSpec and TeamWorkerSpec.
7. Status of the PoC
We have implemented stages 1-5 against an internal HiClaw fork (based on v1.1.2). End-to-end integration tests cover:
| Phase |
Scenario |
Result |
| 1 |
Baseline v1.1.2: legacy POST /teams -> reaches Active |
Pass |
| 2a |
Decoupled controller: Worker CRs exist with no team annotations; Team spec.workerMembers references them |
Pass |
| 2b |
Decoupled POST /teams ({members: [...]}) -> both Worker CRs created, leader replies to user |
Pass |
| 3 |
Legacy POST /teams against the new controller -> reaches Active, then auto-migrator advances to Migrated (workerMembers populated, Worker CRs created, no Pod rebuild) |
Pass |
| 4 |
Conflict detection: duplicate worker name -> 409; mixing both formats -> 400; missing team_leader -> validation error |
Pass |
| 4 |
Three Teams (legacy-baseline, decoupled, legacy-migrated) coexisting and operational on the same cluster |
Pass |
We are happy to share more concrete numbers, screenshots of kubectl describe, and audit logs in this thread if helpful.
8. Discussion points
We would appreciate feedback from maintainers and the community on:
- Annotation-free Worker CRs. Are there use cases (auth, observability, kubectl filtering) that would prefer a thin annotation like
hiclaw.io/team instead of relying purely on the field indexer? We chose the indexer because it makes the Team-Worker relationship single-sourced (Team CR), but we are open to a thin annotation if it eases ops.
- Single-Team-per-Worker for now vs multi-Team-from-day-one. Current design is single-Team-only and would relax later. Is anyone already needing multi-Team membership?
- Migration default. Default
--enable-auto-team-migration=true vs requiring an explicit opt-in. We lean toward "on by default" given the additive, crash-safe design, but understand operators may prefer explicit consent.
- Coordination block format. Today's leader/worker blocks differ in content. We would like to keep that, but if the community prefers a uniform format we are happy to align.
- Deprecation timeline for
LeaderSpec / TeamWorkerSpec. We propose keeping them through one minor release and removing in the next major. Thoughts on the cadence?
If the direction is acceptable, we can split this into a small chain of PRs aligned to the rollout stages above (CRD -> WorkerReconciler -> TeamReconciler new path -> REST/Auth -> Migrator -> cleanup), each independently reviewable and gated by feature flags where it touches behavior.
Thanks for reading this far. Happy to expand any section or share PoC code on request.
Summary
The current
TeamCRD bundles two distinct concerns into a single resource:This bundling has produced concrete pain points: triplicated API types, an over-burdened
TeamReconciler, no independent K8s identity for team members, and tight coupling between organizational changes and runtime state.This issue proposes splitting the two concerns:
TeamCR keeps only organizational/policy fields (workerMembers[],admin,humanMembers,accessPolicy,channelPolicy).WorkerCR, owned byWorkerReconcilerexclusively.TeamReconcilerbecomes a thin coordinator that only manages Matrix Room membership, MinIOAGENTS.mdcoordination injection, and status aggregation. It never touches Pods, ServiceAccounts, configs, or routes.WorkerCRs carry no team annotation. Team membership is expressed only byTeam.spec.workerMembers[].name, queried via field indexer.We have a working PoC of this design (controller + dual-format API + automatic migrator) running in our internal integration environment. End-to-end validation passed across legacy-format, decoupled-format, and mixed scenarios with no Pod rebuilds and no Matrix Room recreation. We would like to upstream this to HiClaw and are opening this issue first to gather feedback on the architecture before sending PRs.
1. Current state (Before)
1.1 The "all-in-one" Team CR
Team members do not have their own
WorkerCR today.TeamReconcilerreads runtime definitions directly fromTeam.spec.leader/Team.spec.workers[]and converges Matrix users, configs, Pods, exposed ports, etc. This means:Workerand a team-memberWorkerare produced by completely different code paths (WorkerReconcilervsTeamReconcilercalling shared phases).1.2 Concrete problems
Problem 1 — Three near-duplicate API types
WorkerSpecTeamWorkerSpecLeaderSpecAdding any new runtime field (e.g.
RemoteSkills) requires changes in all three structs plus theteamWorkerSpecToWorkerSpec()conversion, plus REST request types. Missing one location silently breaks team workers.LeaderSpecis also artificially restricted (noImage, noSkills[]list, noExpose) — those gaps exist purely because keeping three structs in sync is expensive.There is also a
hashMemberSourceSpecinvariant: every new field must useomitemptywith the right zero-value semantics, otherwise upgrading the controller triggers cluster-wide container rebuilds. The code carries a 60-line comment specifically warning about this load-bearing constraint.Problem 2 —
TeamReconcileris too heavyteam_controller.gois over 1000 lines. A single reconcile chains six steps:Steps 2-5 are ~80% of the code and are conceptually about per-member runtime, not team organization. Consequences:
modeltriggers a Team-level reconcile.Degraded.hiclaw.io/teamlabel triggersTeamReconciler.Problem 3 — No independent K8s identity for team members
Team members exist only as elements of
Team.spec.workers[]. There is noWorkerCR for them. As a result:kubectl get worker <team-member>does not work.GET /workers/<team-member>requires server-side synthesis from the Team CR.accessEntriesare scattered at different nesting depths (spec.leader.accessEntries,spec.workers[i].accessEntries).Problem 4 — Organizational changes interfere with runtime state
Adding/removing a member rewrites the same CR that carries every other member's runtime spec. Concretely:
ChannelPolicyripples through every member container.Problem 5 — Dual code paths for runtime work
Every feature touching worker runtime (skills, exposed ports, lifecycle, etc.) has to be wired through two different controllers. Easy to drift, easy to forget one side.
1.3 Summary table
2. Proposed design (After)
2.1 Core principles
WorkerCR has zero team-related annotations or labels.WorkerReconcilernever reads aTeamCR.Team.speclists references to existing Worker CRs and team-level policies — nothing about runtime.WorkerReconcileralone.TeamReconcilerinvites/kicks members in the Matrix Team Room and writes a coordination block into each member's MinIOAGENTS.md. It does not patch the Worker CR or recreate the Pod.2.2 Architecture overview
2.3 New CRD shape
TeamSpec (after):
WorkerSpec / WorkerStatus: unchanged.
Worker.status.matrixUserIDandWorker.status.roomIDalready exist and are read byTeamReconciler.Worker metadata: contains no team-related annotation:
hiclaw.io/team(removed)hiclaw.io/role(removed)hiclaw.io/team-leader(removed)hiclaw.io/team-admin-id(removed)Membership is expressed only via
Team.spec.workerMembers[]and discovered through aspec.workerMembers[*].namefield indexer.Validation rules:
workerMembers[].namemust be unique within a Team.role: team_leaderis required.roledefaults toworker.Degraded.2.4 Reconciler responsibilities
TeamReconciler (after) does only:
Worker.status.matrixUserID, invite/kick).AGENTS.md(different content for leader vs worker — same as today).Worker.status.It no longer: patches Worker CRs, creates Matrix users for workers, deploys configs, creates/recreates Pods, manages exposed routes, or manages ServiceAccounts.
WorkerReconciler (after) is unified for all workers and remains team-agnostic:
Worker.spec.MemberContext(noTeamName, no peer list, no Team channel policy).Worker.status.2.5 Coordination delivery (no Pod rebuild)
Instead of patching CRs to communicate team membership to the agent process, we use two existing channels:
AGENTS.mdinjection —TeamReconcilerwrites a<!-- hiclaw-team-context-start -->...<!-- hiclaw-team-context-end -->block to each member'sagents/<name>/AGENTS.md. The agent picks this up viamc mirror(seconds-level) and hot-reloads.Block content stays role-differentiated, identical in spirit to today:
Adding/removing a member therefore:
workerMembers += {...}).TeamReconcilerinvites the new member's Matrix user to the Team Room.TeamReconcilerwrites/updates MinIO blocks for affected members.No Worker reconcile, no Pod restart.
2.6 Permission resolution
accessPolicy.defaultEntriescarries shared paths (e.g.teams/${self.team}/*,shared/*).spec.accessEntriescarries personal paths (e.g.agents/${self.name}/*).DefaultEntriesForTeamMember()becomes a fallback used only when a Team has no explicitaccessPolicy.2.7 REST API surface
Workers:
POST /workersGET /workers/{name}GET /workersPUT /workers/{name}DELETE /workers/{name}Teams:
POST /teamsPUT /teams/{name}DELETE /teams/{name}2.8 Dual-format
POST /teamsduring the migration windowTo keep existing clients working, the create-team handler accepts both shapes:
{leader: {...}, workers: [...]}-> converted into Worker CRs +workerMembers[]automatically.{members: [{name, role, ...}, ...]}-> direct path.Sending both at the same time is a 400 error. Validation includes "exactly one team_leader" and "no duplicate worker names".
3. Automatic migration (in-controller)
Existing Team CRs are migrated by the controller itself — no external CLI, no Pod rebuilds, no Matrix Room recreation.
3.1 State machine
Each transition writes
team.metadata.annotations["hiclaw.io/migration-phase"]so a crashed/restarted controller can resume.NotStarted -> WorkerCRsCreatedhiclaw.io/migration-in-flighton TeamWorkerCRsCreated -> StatusSeededWorker.status(matrixUserID, roomID) so existing Matrix users / rooms stay bound to the new CRStatusSeeded -> PodsReparentedmetadata.ownerReferencesto point to its Worker CRPodsReparented -> CoordinationInjectedCoordinationInjected -> TeamSpecPatchedspec.workerMembers[]andspec.accessPolicy; oldleader/workersfields keptTeamSpecPatched -> Migrated3.2 Safety gates
--enable-auto-team-migration=true(default on).hiclaw.io/auto-migrate=disabledto opt out, or=rollbackto reverse.phase=Activeand every member has non-emptyMatrixUserIDandRoomID.3.3 "Move-then-write" atomicity
The new CRD schema drops legacy
TeamMemberStatusfields (Observed,SpecHash,RoomID,ExposedPorts,RuntimeName) outright. We rely on K8s pruning semantics: pruning happens on the next mutating write, not on read. The migrator therefore enforces this order per Team:GETTeam CR (reads legacyStatus.Memberswhile it still exists in etcd).Worker.statusfrom the legacy values.UPDATEthe Team CR with the new shape — this is when pruning fires.A crash before step 5 leaves the Team CR untouched and the Worker CRs idempotent; the migrator simply resumes.
3.4 Pod ownerReference reparent
Today's team-member Pods have
ownerReferences[*].controllerpointing at the Team CR, sokubectl delete teamcascades into Pods. Post-migration the owner must be the Worker CR; otherwise:The reparent uses a single JSON Patch and is guarded by the
hiclaw.io/migration-in-flightfinalizer to prevent a delete during the window.3.5 Observability
Normal MigrationPhaseAdvanced/Warning MigrationPhaseFailedper transition.hiclaw_team_migration_phase{team,phase}gauge,hiclaw_team_migration_failures_totalcounter.kubectl get teamgains aMIGRATIONprinter column.4. Acceptance criteria
A reasonable definition of "done" for the upstream change:
hiclaw.io/team,hiclaw.io/role, orhiclaw.io/team-leaderannotations.WorkerReconcilerreconcile events for affected workers.spec.accessEntriesand TeamaccessPolicy.defaultEntries.workerMembers) keeps working via the legacy reconcile path.Migratedwith no Pod or Matrix Room recreation.ownerReferenceuniquely points at its Worker CR.hiclaw.io/auto-migrate=rollbackreverses to the legacy shape.5. Risks and mitigations
Team.status.message; agent degrades to standalonemc mirrorlag delays team-context deliveryAGENTS.mdWorkerdeleted while still referenced byTeam.workerMembersDegradedwith a clear message6. Suggested rollout order
AGENTS.mdcoordination blocks without restart.WorkerMembersandAccessPolicytoTeamSpec(deprecateLeader/Workersbut keep them). No behavior change.WorkerReconciler.MemberContextis always standalone. Standalone Workers are unaffected.TeamReconcilertakes the new path whenspec.WorkerMembers != nil: validate -> infra -> invite -> MinIO inject -> status. No shared phases called.CoordinationInjectedphase; safety gates and rollback.buildDesiredMembers/reconcileMemberpath and (in a major version) dropLeaderSpecandTeamWorkerSpec.7. Status of the PoC
We have implemented stages 1-5 against an internal HiClaw fork (based on v1.1.2). End-to-end integration tests cover:
POST /teams-> reachesActivespec.workerMembersreferences themPOST /teams({members: [...]}) -> both Worker CRs created, leader replies to userPOST /teamsagainst the new controller -> reachesActive, then auto-migrator advances toMigrated(workerMembers populated, Worker CRs created, no Pod rebuild)team_leader-> validation errorWe are happy to share more concrete numbers, screenshots of
kubectl describe, and audit logs in this thread if helpful.8. Discussion points
We would appreciate feedback from maintainers and the community on:
hiclaw.io/teaminstead of relying purely on the field indexer? We chose the indexer because it makes the Team-Worker relationship single-sourced (Team CR), but we are open to a thin annotation if it eases ops.--enable-auto-team-migration=truevs requiring an explicit opt-in. We lean toward "on by default" given the additive, crash-safe design, but understand operators may prefer explicit consent.LeaderSpec/TeamWorkerSpec. We propose keeping them through one minor release and removing in the next major. Thoughts on the cadence?If the direction is acceptable, we can split this into a small chain of PRs aligned to the rollout stages above (CRD -> WorkerReconciler -> TeamReconciler new path -> REST/Auth -> Migrator -> cleanup), each independently reviewable and gated by feature flags where it touches behavior.
Thanks for reading this far. Happy to expand any section or share PoC code on request.