Skip to content

[Proposal] Decouple Team and Worker CRs: separate organizational membership from runtime definition #902

Description

@Sunrisea

Summary

The current Team CRD bundles two distinct concerns into a single resource:

  1. Organizational relationships — who is in the team, who is the leader, who is admin, peer-mention policy.
  2. Per-member runtime definition — model, identity, soul, package, mcpServers, env, accessEntries, etc. for every leader/worker.

This bundling has produced concrete pain points: triplicated API types, an over-burdened TeamReconciler, no independent K8s identity for team members, and tight coupling between organizational changes and runtime state.

This issue proposes splitting the two concerns:

  • Team CR keeps only organizational/policy fields (workerMembers[], admin, humanMembers, accessPolicy, channelPolicy).
  • Every team member becomes a regular Worker CR, owned by WorkerReconciler exclusively.
  • TeamReconciler becomes a thin coordinator that only manages Matrix Room membership, MinIO AGENTS.md coordination injection, and status aggregation. It never touches Pods, ServiceAccounts, configs, or routes.
  • Crucially, Worker CRs carry no team annotation. Team membership is expressed only by Team.spec.workerMembers[].name, queried via field indexer.

We have a working PoC of this design (controller + dual-format API + automatic migrator) running in our internal integration environment. End-to-end validation passed across legacy-format, decoupled-format, and mixed scenarios with no Pod rebuilds and no Matrix Room recreation. We would like to upstream this to HiClaw and are opening this issue first to gather feedback on the architecture before sending PRs.


1. Current state (Before)

1.1 The "all-in-one" Team CR

type TeamSpec struct {
    Description   string             `json:"description,omitempty"`
    TeamName      string             `json:"teamName,omitempty"`
    Admin         *TeamAdminSpec     `json:"admin,omitempty"`
    HumanMembers  []TeamMemberSpec   `json:"humanMembers,omitempty"`
    Leader        LeaderSpec         `json:"leader"`              // 15 runtime fields inlined
    Workers       []TeamWorkerSpec   `json:"workers,omitempty"`   // 18 runtime fields inlined per member
    PeerMentions  *bool              `json:"peerMentions,omitempty"`
    ChannelPolicy *ChannelPolicySpec `json:"channelPolicy,omitempty"`
}

Team members do not have their own Worker CR today. TeamReconciler reads runtime definitions directly from Team.spec.leader / Team.spec.workers[] and converges Matrix users, configs, Pods, exposed ports, etc. This means:

  • "Who belongs to the team" and "how that member runs" are the same Kubernetes object.
  • A standalone Worker and a team-member Worker are produced by completely different code paths (WorkerReconciler vs TeamReconciler calling shared phases).

1.2 Concrete problems

Problem 1 — Three near-duplicate API types

struct fields purpose
WorkerSpec 18 standalone Worker runtime
TeamWorkerSpec 18 team worker runtime (≈1:1 duplicate)
LeaderSpec 15 team leader runtime (subset of WorkerSpec)

Adding any new runtime field (e.g. RemoteSkills) requires changes in all three structs plus the teamWorkerSpecToWorkerSpec() conversion, plus REST request types. Missing one location silently breaks team workers. LeaderSpec is also artificially restricted (no Image, no Skills[] list, no Expose) — those gaps exist purely because keeping three structs in sync is expensive.

There is also a hashMemberSourceSpec invariant: every new field must use omitempty with the right zero-value semantics, otherwise upgrading the controller triggers cluster-wide container rebuilds. The code carries a 60-line comment specifically warning about this load-bearing constraint.

Problem 2 — TeamReconciler is too heavy

team_controller.go is over 1000 lines. A single reconcile chains six steps:

reconcileTeamNormal:
  1. Provision Team Room / Leader DM / Shared Storage     (team infra)
  2. Write inline configs (Identity/Soul/Agents to MinIO) (member runtime)
  3. Cleanup stale members                                (member runtime)
  3.5 Inject coordination context                         (member runtime)
  4. For each member -> shared phases:                    (member runtime, the bulk)
       Matrix user / SA / Config / Container / Expose
  5. Update legacy registry                               (member metadata)
  6. Aggregate status                                     (status)

Steps 2-5 are ~80% of the code and are conceptually about per-member runtime, not team organization. Consequences:

  • Changing one member's model triggers a Team-level reconcile.
  • A single backend failure tips the whole Team into Degraded.
  • Pod watch is broad: every Pod with a hiclaw.io/team label triggers TeamReconciler.
  • Debugging a worker startup means reading Team reconcile logs.

Problem 3 — No independent K8s identity for team members

Team members exist only as elements of Team.spec.workers[]. There is no Worker CR for them. As a result:

  • kubectl get worker <team-member> does not work.
  • GET /workers/<team-member> requires server-side synthesis from the Team CR.
  • The auth enricher needs a field indexer to reverse-look-up Team CRs because there is no Worker CR to read.
  • accessEntries are scattered at different nesting depths (spec.leader.accessEntries, spec.workers[i].accessEntries).

Problem 4 — Organizational changes interfere with runtime state

Adding/removing a member rewrites the same CR that carries every other member's runtime spec. Concretely:

  • Adding a new member triggers communication-policy and hash recomputation for every existing member.
  • Changing ChannelPolicy ripples through every member container.
  • Deleting the Team is effectively "delete all members' runtime."
  • Editing one member's model triggers a Team-level reconcile.

Problem 5 — Dual code paths for runtime work

Every feature touching worker runtime (skills, exposed ports, lifecycle, etc.) has to be wired through two different controllers. Easy to drift, easy to forget one side.

1.3 Summary table

Domain Issue Root cause Impact
API design Three duplicate types Team CR embeds runtime defs High evolution cost; Leader feature-gapped
Controller arch TeamReconciler over-burdened Org + runtime in one loop Whole-team coupling, hard debugging, no per-member retry
Resource model No member-level CR Inline design choice Standard kubectl unusable, API hacks, indexer-based auth
Concerns Org change disturbs runtime Single CR, two responsibilities Member changes are destructive
Engineering Two code paths Two controller entry points Low velocity, drift risk

2. Proposed design (After)

2.1 Core principles

  1. Workers are team-agnostic. A Worker CR has zero team-related annotations or labels. WorkerReconciler never reads a Team CR.
  2. Team is organization + policy only. Team.spec lists references to existing Worker CRs and team-level policies — nothing about runtime.
  3. Every agent has a Worker CR. Standalone or team member, runtime is owned by WorkerReconciler alone.
  4. Team semantics are delivered out-of-band. TeamReconciler invites/kicks members in the Matrix Team Room and writes a coordination block into each member's MinIO AGENTS.md. It does not patch the Worker CR or recreate the Pod.
  5. Pod ownerReference points to the Worker CR, not the Team.
  6. Permissions are merged at issuance time — Worker.spec.accessEntries union Team.spec.accessPolicy.defaultEntries.
  7. Migration is gradual and additive — old Team CRs remain functional during a compatibility window; a controller-internal migrator advances them.

2.2 Architecture overview

+---------------------------------------------------------------------+
|                     User / Control-plane API                        |
+---------------------------------------------------------------------+
|  POST /workers  ->  Worker CR (runtime definition)                  |
|  POST /teams    ->  Team CR   (workerMembers[] + accessPolicy + ...)|
+----------+---------------------------------------+------------------+
           |                                       |
    +------v------+                          +-----v------+
    |  Worker CR  |   <- TeamReconciler      |  Team CR   |
    | (runtime)   |     reads Worker.status  | (org+policy|
    | no team ann |     only                 |     )      |
    +------+------+                          +-----+------+
           |                                       |
    +------v---------+                       +-----v-----------+
    | WorkerRecon    |                       | TeamReconciler  |
    | team-agnostic  |                       | out-of-band     |
    | Pod / config / |                       | delivery only:  |
    | matrix user /  |                       |   - Matrix      |
    | expose         |                       |     invite/kick |
    +----------------+                       |   - MinIO       |
                                             |     AGENTS.md   |
                                             |   - status agg  |
                                             +-----------------+

2.3 New CRD shape

TeamSpec (after):

type TeamSpec struct {
    Description   string             `json:"description,omitempty"`
    TeamName      string             `json:"teamName,omitempty"`
    Admin         *TeamAdminSpec     `json:"admin,omitempty"`
    HumanMembers  []TeamMemberSpec   `json:"humanMembers,omitempty"`

    // Reference existing Worker CRs (replaces inlined runtime definitions).
    WorkerMembers []TeamWorkerRef    `json:"workerMembers,omitempty"`

    // Team-level access policy (replaces hard-coded DefaultEntriesForTeamMember).
    AccessPolicy  *TeamAccessPolicySpec `json:"accessPolicy,omitempty"`

    PeerMentions  *bool              `json:"peerMentions,omitempty"`
    ChannelPolicy *ChannelPolicySpec `json:"channelPolicy,omitempty"`

    // Deprecated -- kept during the compatibility window. Ignored when WorkerMembers is set.
    Leader  LeaderSpec       `json:"leader,omitempty"`
    Workers []TeamWorkerSpec `json:"workers,omitempty"`
}

type TeamWorkerRef struct {
    Name string `json:"name"`           // metadata.name of the referenced Worker CR
    Role string `json:"role,omitempty"` // "team_leader" | "worker" (default "worker")
}

type TeamAccessPolicySpec struct {
    // Merged with Worker.spec.accessEntries at credential issuance time.
    DefaultEntries []AccessEntry `json:"defaultEntries,omitempty"`
}

WorkerSpec / WorkerStatus: unchanged. Worker.status.matrixUserID and Worker.status.roomID already exist and are read by TeamReconciler.

Worker metadata: contains no team-related annotation:

  • hiclaw.io/team (removed)
  • hiclaw.io/role (removed)
  • hiclaw.io/team-leader (removed)
  • hiclaw.io/team-admin-id (removed)

Membership is expressed only via Team.spec.workerMembers[] and discovered through a spec.workerMembers[*].name field indexer.

Validation rules:

  • workerMembers[].name must be unique within a Team.
  • Exactly one role: team_leader is required.
  • Empty role defaults to worker.
  • A missing referenced Worker CR makes the Team Degraded.
  • Initial constraint: a Worker may belong to at most one Team. This is enforced via the field indexer (multiple hits -> error). Multi-Team is not blocked by design and can be relaxed later.

2.4 Reconciler responsibilities

TeamReconciler (after) does only:

  1. Validate membership (refs exist, exactly one leader, no duplicates).
  2. Provision team infrastructure (Team Room, Leader DM, shared storage).
  3. Manage Matrix room membership (read Worker.status.matrixUserID, invite/kick).
  4. Inject coordination block into every member's MinIO AGENTS.md (different content for leader vs worker — same as today).
  5. Update legacy registries.
  6. Aggregate status from Worker.status.

It no longer: patches Worker CRs, creates Matrix users for workers, deploys configs, creates/recreates Pods, manages exposed routes, or manages ServiceAccounts.

WorkerReconciler (after) is unified for all workers and remains team-agnostic:

  1. Read Worker.spec.
  2. Build a standalone MemberContext (no TeamName, no peer list, no Team channel policy).
  3. Run shared phases (Infra, ServiceAccount, Config, Container, Expose).
  4. Write Worker.status.

2.5 Coordination delivery (no Pod rebuild)

Instead of patching CRs to communicate team membership to the agent process, we use two existing channels:

  • Matrix Room invite/kick — agents auto-accept invites via Matrix sync; getting kicked is observable as a room-leave event.
  • MinIO AGENTS.md injectionTeamReconciler writes a <!-- hiclaw-team-context-start --> ... <!-- hiclaw-team-context-end --> block to each member's agents/<name>/AGENTS.md. The agent picks this up via mc mirror (seconds-level) and hot-reloads.

Block content stays role-differentiated, identical in spirit to today:

  • Leader gets the full picture: team room, leader DM, worker list, heartbeat/idle timeouts, behavioral rules.
  • Worker gets only its coordinator + admin + behavioral rules — no peer information.

Adding/removing a member therefore:

  1. Patches the Team CR (workerMembers += {...}).
  2. TeamReconciler invites the new member's Matrix user to the Team Room.
  3. TeamReconciler writes/updates MinIO blocks for affected members.

No Worker reconcile, no Pod restart.

2.6 Permission resolution

func (r *Resolver) ResolveForCaller(ctx context.Context, caller *CallerIdentity) (...) {
    workerEntries := r.readWorkerEntries(ctx, caller.Username) // always
    if caller.Team != "" {
        teamEntries := r.readTeamDefaultEntries(ctx, caller.Team)
        if teamEntries == nil {
            teamEntries = DefaultEntriesForTeamMember(caller.Username, caller.Team) // fallback
        }
        return union(workerEntries, teamEntries), nil
    }
    return workerEntries, nil
}
  • Team accessPolicy.defaultEntries carries shared paths (e.g. teams/${self.team}/*, shared/*).
  • Worker spec.accessEntries carries personal paths (e.g. agents/${self.name}/*).
  • Union, no conflict resolution needed (paths are namespaced).
  • DefaultEntriesForTeamMember() becomes a fallback used only when a Team has no explicit accessPolicy.

2.7 REST API surface

Workers:

Endpoint Before After
POST /workers rejects names that match team members always allowed; team members are also Workers
GET /workers/{name} synthesizes from Team CR for team members reads Worker CR directly
GET /workers merges standalone + synthesized lists Worker CRs directly
PUT /workers/{name} rejects team members allowed
DELETE /workers/{name} rejects team members allowed (also drops the Team reference)

Teams:

Endpoint Before After
POST /teams full inline runtime only org + policy fields
PUT /teams/{name} edits everything edits only org + policy
DELETE /teams/{name} cascades into all members tears down team infra; does not delete Workers

2.8 Dual-format POST /teams during the migration window

To keep existing clients working, the create-team handler accepts both shapes:

  • Legacy: {leader: {...}, workers: [...]} -> converted into Worker CRs + workerMembers[] automatically.
  • Decoupled: {members: [{name, role, ...}, ...]} -> direct path.

Sending both at the same time is a 400 error. Validation includes "exactly one team_leader" and "no duplicate worker names".


3. Automatic migration (in-controller)

Existing Team CRs are migrated by the controller itself — no external CLI, no Pod rebuilds, no Matrix Room recreation.

3.1 State machine

NotStarted -> WorkerCRsCreated -> StatusSeeded -> PodsReparented
                                                        |
                                                        v
   Migrated <- TeamSpecPatched <- CoordinationInjected

Each transition writes team.metadata.annotations["hiclaw.io/migration-phase"] so a crashed/restarted controller can resume.

Transition Action
NotStarted -> WorkerCRsCreated Create Worker CRs for the leader and each worker; add finalizer hiclaw.io/migration-in-flight on Team
WorkerCRsCreated -> StatusSeeded Patch Worker.status (matrixUserID, roomID) so existing Matrix users / rooms stay bound to the new CR
StatusSeeded -> PodsReparented JSON-Patch each member Pod's metadata.ownerReferences to point to its Worker CR
PodsReparented -> CoordinationInjected Matrix invite all members + write MinIO coordination blocks
CoordinationInjected -> TeamSpecPatched Write spec.workerMembers[] and spec.accessPolicy; old leader/workers fields kept
TeamSpecPatched -> Migrated Remove finalizer; mark migrated

3.2 Safety gates

  • Cluster flag --enable-auto-team-migration=true (default on).
  • Per-Team annotation hiclaw.io/auto-migrate=disabled to opt out, or =rollback to reverse.
  • Health pre-check: Team phase=Active and every member has non-empty MatrixUserID and RoomID.
  • Token bucket on starting migrations (1 QPS, concurrency 1 by default). Once a Team enters the state machine it is not rate-limited — half-migrated states are not allowed to linger.
  • Leader-elected: only the leader controller runs migrator steps.

3.3 "Move-then-write" atomicity

The new CRD schema drops legacy TeamMemberStatus fields (Observed, SpecHash, RoomID, ExposedPorts, RuntimeName) outright. We rely on K8s pruning semantics: pruning happens on the next mutating write, not on read. The migrator therefore enforces this order per Team:

  1. GET Team CR (reads legacy Status.Members while it still exists in etcd).
  2. Create Worker CRs and seed Worker.status from the legacy values.
  3. Reparent Pods.
  4. Inject coordination.
  5. Last step: UPDATE the Team CR with the new shape — this is when pruning fires.

A crash before step 5 leaves the Team CR untouched and the Worker CRs idempotent; the migrator simply resumes.

3.4 Pod ownerReference reparent

Today's team-member Pods have ownerReferences[*].controller pointing at the Team CR, so kubectl delete team cascades into Pods. Post-migration the owner must be the Worker CR; otherwise:

  • Deleting the Team would still nuke worker Pods.
  • Deleting the Worker would leave Pods orphaned.

The reparent uses a single JSON Patch and is guarded by the hiclaw.io/migration-in-flight finalizer to prevent a delete during the window.

3.5 Observability

  • Events: Normal MigrationPhaseAdvanced / Warning MigrationPhaseFailed per transition.
  • Metrics: hiclaw_team_migration_phase{team,phase} gauge, hiclaw_team_migration_failures_total counter.
  • kubectl get team gains a MIGRATION printer column.

4. Acceptance criteria

A reasonable definition of "done" for the upstream change:

  1. No Worker CR carries hiclaw.io/team, hiclaw.io/role, or hiclaw.io/team-leader annotations.
  2. Adding/removing a Team member produces zero WorkerReconciler reconcile events for affected workers.
  3. Pod restart count is unchanged across membership changes.
  4. Coordination context propagates within seconds (mc mirror cadence) after a Team change.
  5. Issued STS tokens contain the union of Worker spec.accessEntries and Team accessPolicy.defaultEntries.
  6. Removing a Worker from a Team leaves the Worker CR + Pod intact and switches the agent to standalone mode automatically.
  7. Deleting a Team CR cascades into Team infrastructure only; no Worker Pod is deleted.
  8. A pre-existing Team CR (no workerMembers) keeps working via the legacy reconcile path.
  9. Auto-migration drives every legacy Team to Migrated with no Pod or Matrix Room recreation.
  10. Post-migration, every member Pod's ownerReference uniquely points at its Worker CR.
  11. hiclaw.io/auto-migrate=rollback reverses to the legacy shape.
  12. Killing the controller mid-migration is safe; the next leader resumes from the last persisted phase.

5. Risks and mitigations

Risk Mitigation
MinIO write fails -> member misses team context TeamReconciler retries; surface in Team.status.message; agent degrades to standalone
mc mirror lag delays team-context delivery Mirror cadence is seconds; agents also use Matrix room-join events as a secondary signal
Field indexer cost at very large Team counts Backed by informer cache, O(1) lookup; revisit at 1000+ Teams
Worker referenced by multiple Teams (not initially supported) Indexer returns multiple hits -> enricher errors; admission webhook can also reject
Old agents that don't hot-reload AGENTS.md Restart fallback works; document minimum agent version
Users assume "delete Team" stops workers Documented + UI hint; semantics: Workers continue independently
Worker deleted while still referenced by Team.workerMembers Team flips to Degraded with a clear message
Pod reparent fails Atomic JSON Patch + exponential backoff + finalizer blocks Team deletion
State-machine failure loop Backoff + per-Team disable annotation + alerting metrics
Annotation patch failure on Worker Not applicable — this design never patches Worker annotations

6. Suggested rollout order

Stage 0   ->  Stage 1   ->  Stage 2          ->  Stage 3        ->  Stage 4    ->  Stage 5    ->  Stage 6
Agent         CRD           WorkerReconciler     TeamReconciler     REST +         Auto-          Drop
hot-reload    extension     becomes              new path           Auth/Access    migration      legacy
prereq        (no behavior  team-agnostic                                                         path
              change)
  • Stage 0: Confirm the agent runtime can hot-reload AGENTS.md coordination blocks without restart.
  • Stage 1: Add WorkerMembers and AccessPolicy to TeamSpec (deprecate Leader/Workers but keep them). No behavior change.
  • Stage 2: Strip team-annotation reads from WorkerReconciler. MemberContext is always standalone. Standalone Workers are unaffected.
  • Stage 3: TeamReconciler takes the new path when spec.WorkerMembers != nil: validate -> infra -> invite -> MinIO inject -> status. No shared phases called.
  • Stage 4: Dual-format REST handler; CREnricher uses field indexer only; AccessResolver merges entries.
  • Stage 5: Migrator state machine with the CoordinationInjected phase; safety gates and rollback.
  • Stage 6: Remove the legacy buildDesiredMembers/reconcileMember path and (in a major version) drop LeaderSpec and TeamWorkerSpec.

7. Status of the PoC

We have implemented stages 1-5 against an internal HiClaw fork (based on v1.1.2). End-to-end integration tests cover:

Phase Scenario Result
1 Baseline v1.1.2: legacy POST /teams -> reaches Active Pass
2a Decoupled controller: Worker CRs exist with no team annotations; Team spec.workerMembers references them Pass
2b Decoupled POST /teams ({members: [...]}) -> both Worker CRs created, leader replies to user Pass
3 Legacy POST /teams against the new controller -> reaches Active, then auto-migrator advances to Migrated (workerMembers populated, Worker CRs created, no Pod rebuild) Pass
4 Conflict detection: duplicate worker name -> 409; mixing both formats -> 400; missing team_leader -> validation error Pass
4 Three Teams (legacy-baseline, decoupled, legacy-migrated) coexisting and operational on the same cluster Pass

We are happy to share more concrete numbers, screenshots of kubectl describe, and audit logs in this thread if helpful.


8. Discussion points

We would appreciate feedback from maintainers and the community on:

  1. Annotation-free Worker CRs. Are there use cases (auth, observability, kubectl filtering) that would prefer a thin annotation like hiclaw.io/team instead of relying purely on the field indexer? We chose the indexer because it makes the Team-Worker relationship single-sourced (Team CR), but we are open to a thin annotation if it eases ops.
  2. Single-Team-per-Worker for now vs multi-Team-from-day-one. Current design is single-Team-only and would relax later. Is anyone already needing multi-Team membership?
  3. Migration default. Default --enable-auto-team-migration=true vs requiring an explicit opt-in. We lean toward "on by default" given the additive, crash-safe design, but understand operators may prefer explicit consent.
  4. Coordination block format. Today's leader/worker blocks differ in content. We would like to keep that, but if the community prefers a uniform format we are happy to align.
  5. Deprecation timeline for LeaderSpec / TeamWorkerSpec. We propose keeping them through one minor release and removing in the next major. Thoughts on the cadence?

If the direction is acceptable, we can split this into a small chain of PRs aligned to the rollout stages above (CRD -> WorkerReconciler -> TeamReconciler new path -> REST/Auth -> Migrator -> cleanup), each independently reviewable and gated by feature flags where it touches behavior.

Thanks for reading this far. Happy to expand any section or share PoC code on request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:worker-runtimeManager/Worker 运行时、CoPaw/QwenPaw、Worker 容器、技能同步、Worker 启停、Manager-Worker 运行链路、本地状态与配置持久化

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions