Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Hive did not historically maintain a complete changelog. This file starts a prag
- Linear work sources now preserve GitHub-path parity without changing GitHub/GHE enumeration: Linear items carry age, priority/state, tracker/parent signals, project routing, current-cycle filtering, GitHub-style hold matching, exempt-label filtering, and SLA summaries through the existing actionable queue. Invalid Linear work-source configs fail closed instead of idling the fleet as an empty backlog, and standalone Docker/Quadlet deployments pass the Linear environment variables needed to use the integration ([#4722](https://github.com/kubestellar/hive/issues/4722)).
- Trusted-tier agents can push branches that modify `.github/workflows/` once the GitHub App grants **Workflows: Read and write** (#4677): scoped token minting for the `trusted`/`merger` tiers now requests `workflows:write`, with an honest fallback — on installations that have not accepted the permission yet, the mint retries without it and logs exactly what the operator must flip instead of breaking every trusted-tier token. The App setup doc's permission table gained the Workflows row (including the re-accept-per-installation step); contributor and advisory tiers deliberately still never request it, keeping the pushbroker's protected-path boundary meaningful.
- Two more stale-branch references in the docs are gone: `examples/kubestellar/README.md` now opens with a "Historical v1 case study" callout (matching `docs/LEGACY-v1-architecture.md`'s labeling) so new operators are not led into the retired launchd/`worker.sh` install path, and `src/policies/README.md` no longer says "v2 HEAD" for behavior that is current v4 — v2 was retired in August 2026.
- Applying an ACMM level now explicitly reports every governor evaluation-interval and per-agent cadence change. The pack-apply API returns exact before/after values, the dashboard shows them immediately, and the hive writes a WARN log entry, so an operator is not surprised by a different schedule when the governor next restarts ([#4711](https://github.com/kubestellar/hive/issues/4711)).
- The quadlet-gate CI workflow pins `quay.io/podman/stable` by index digest, and quay.io garbage-collected the pinned 2026-08-21 index again — every branch's gate failed with "manifest unknown" before any repo code ran. Re-pinned to the replacement index (`sha256:3b9f8367…`), which still carries podman 5.8.4 per its OCI version label, so the generator under test is unchanged.
- The scanner example policy (`examples/kubestellar/agents/scanner.md`) no longer dead-links its memory files: the six `feedback_*.md` files it references (`manual_login_only`, `fix_enhancements_too`, `security_screening`, `scanner_check_existing`, `verify_issues_before_fixing`, `auto_issues`) and the twice-referenced `project_ci-maintainer_policy.md` now exist beside it with the operator-facing policy content the links promise, derived from the behaviour scanner.md itself specifies ([#4635](https://github.com/kubestellar/hive/issues/4635)).
- Every per-UID CLI agent now runs with its **own HOME** (`/data/home/agents/<name>`) instead of the fleet sharing `/data/home`, resolving the one-agent-signs-out-everyone defect ([#4596](https://github.com/kubestellar/hive/issues/4596)). The mechanism, measured on a live hive: Claude Code rewrites `$HOME/.claude.json` wholesale via atomic rename, and rename needs only *directory* write permission — the shared `2775 dev:node` home let any agent replace the file regardless of its mode, so an unauthenticated agent's rewrite stripped `oauthAccount` from under the authenticated one and the whole fleet fell back to the "Select login method" menu. File permissions can never close a directory-entry race, which is why [#4606](https://github.com/kubestellar/hive/pull/4606) shipped diagnosis and a bounded restart rather than a permission fix. Each per-agent home is symlink-bridged back to the shared state that is safe and useful to share — `.claude` (holding `.credentials.json`, the OAuth token, so **one interactive login still authenticates the fleet**), `.copilot`, `.config`, `.codex`, `.bob`, `.gemini`, caches, and the entrypoint-written `.gitconfig`/`.bashrc`/`.profile` — while the contended session file becomes truly per-agent and is *adopted* at launch from a signed-in source (the legacy shared file, or a signed-in sibling), never fabricated: with no signed-in source anywhere the login menu is the honest state and still appears. This also makes #4606's capped token-triggered restart curative — a restarted agent re-provisions and adopts the sibling session the operator just signed in. Existing hives migrate with **no operator step**: credentials stay where they are, the first launch adopts the legacy session, orphaned `.claude.json.tmp.*` debris in the shared home is swept, and `.bash_history` contention disappears as a side effect. `HIVE_SHARED_AGENT_HOME=1` restores the legacy shared layout wholesale if anything needs the old shape.
Expand Down
89 changes: 75 additions & 14 deletions src/pkg/dashboard/api_packs.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package dashboard
import (
"fmt"
"net/http"
"sort"
"strconv"
"strings"
"sync"
Expand Down Expand Up @@ -42,18 +43,43 @@ func (s *Server) handlePacksList(w http.ResponseWriter, r *http.Request) {

// ApplyPackResult holds the outcome of applying an ACMM pack.
type ApplyPackResult struct {
Name string `json:"name"`
Created []string `json:"created"`
Updated []string `json:"updated"`
Skipped []string `json:"skipped"`
Paused []string `json:"paused"`
Resumed []string `json:"resumed"`
Name string `json:"name"`
Created []string `json:"created"`
Updated []string `json:"updated"`
Skipped []string `json:"skipped"`
Paused []string `json:"paused"`
Resumed []string `json:"resumed"`
GovernorChanges *GovernorChanges `json:"governor_changes,omitempty"`
// Tombstoned lists pack agents deliberately deleted by the operator and
// therefore NOT re-created. Surfaced so an under-full roster reads as an
// honored choice rather than an apply that quietly dropped agents.
Tombstoned []string `json:"tombstoned,omitempty"`
}

// GovernorChanges reports the governor settings a pack apply actually changed.
// It captures old values before reconciliation, because a running governor can
// retain its current ticker until the hive restarts.
type GovernorChanges struct {
EvalIntervalS *GovernorIntervalChange `json:"eval_interval_s,omitempty"`
Cadences []GovernorCadenceChange `json:"cadences,omitempty"`
}

type GovernorIntervalChange struct {
From int `json:"from"`
To int `json:"to"`
}

type GovernorCadenceChange struct {
Mode string `json:"mode"`
Agent string `json:"agent"`
From string `json:"from"`
To string `json:"to"`
}

func (g *GovernorChanges) empty() bool {
return g == nil || (g.EvalIntervalS == nil && len(g.Cadences) == 0)
}

// ApplyPack applies the ACMM pack for the given level. It creates agents,
// sets governor config (eval interval, cadences, thresholds, stale timeouts),
// syncs agent visibility, and persists state. Callable from both the HTTP
Expand Down Expand Up @@ -256,9 +282,15 @@ func (s *Server) applyPack(level int, forceGovernor bool) (*ApplyPackResult, err
// agent) kept the previous level's cadences — including a stale SURGE=pause —
// so the hive stopped kicking agents after a level switch.
isFirstApplyOrExpansion := len(created) > 0 || forceGovernor
governorChanges := &GovernorChanges{}

if pack.Governor.EvalIntervalS > 0 && isFirstApplyOrExpansion {
s.deps.Config.Governor.EvalIntervalS = pack.Governor.EvalIntervalS
from := s.deps.Config.Governor.EvalIntervalS
to := pack.Governor.EvalIntervalS
if from != to {
governorChanges.EvalIntervalS = &GovernorIntervalChange{From: from, To: to}
s.deps.Config.Governor.EvalIntervalS = to
}
}

if len(pack.Governor.Cadences) > 0 || len(pack.Governor.Thresholds) > 0 {
Expand All @@ -272,7 +304,13 @@ func (s *Server) applyPack(level int, forceGovernor bool) (*ApplyPackResult, err
}
for agent, interval := range agentCadences {
if isFirstApplyOrExpansion {
mode.Cadences[agent] = config.Cadence(interval)
from := string(mode.Cadences[agent])
if from != interval {
governorChanges.Cadences = append(governorChanges.Cadences, GovernorCadenceChange{
Mode: modeName, Agent: agent, From: from, To: interval,
})
mode.Cadences[agent] = config.Cadence(interval)
}
} else if _, exists := mode.Cadences[agent]; !exists {
mode.Cadences[agent] = config.Cadence(interval)
}
Expand Down Expand Up @@ -301,6 +339,12 @@ func (s *Server) applyPack(level int, forceGovernor bool) (*ApplyPackResult, err
}
}
}
sort.Slice(governorChanges.Cadences, func(i, j int) bool {
if governorChanges.Cadences[i].Mode == governorChanges.Cadences[j].Mode {
return governorChanges.Cadences[i].Agent < governorChanges.Cadences[j].Agent
}
return governorChanges.Cadences[i].Mode < governorChanges.Cadences[j].Mode
})

for _, pa := range pack.Agents {
if pa.StaleTimeout > 0 {
Expand All @@ -322,6 +366,16 @@ func (s *Server) applyPack(level int, forceGovernor bool) (*ApplyPackResult, err

s.persistOnly()
go s.refreshAsync()
if !governorChanges.empty() {
logArgs := []any{"hive_id", s.deps.Config.HiveID, "level", level, "name", pack.Name}
if interval := governorChanges.EvalIntervalS; interval != nil {
logArgs = append(logArgs, "governor_eval_interval_s_from", interval.From, "governor_eval_interval_s_to", interval.To)
}
if len(governorChanges.Cadences) > 0 {
logArgs = append(logArgs, "cadence_changes", governorChanges.Cadences)
}
s.logger.Warn("ACMM pack changed governor settings", logArgs...)
}
// Observability (#2439): the counts alone ("tombstoned":0) were the tell in the
// field report, so also list WHICH agents were tombstoned (deleted by the operator
// and NOT re-created) vs skipped (already present) — a grep by agent name against
Expand All @@ -344,6 +398,9 @@ func (s *Server) applyPack(level int, forceGovernor bool) (*ApplyPackResult, err
Resumed: resumed,
Tombstoned: tombstoned,
}
if !governorChanges.empty() {
result.GovernorChanges = governorChanges
}
if len(createErrs) > 0 {
// Return the result alongside the error so callers can still see what
// was reconciled, but the non-nil error signals the roster is not
Expand Down Expand Up @@ -388,6 +445,9 @@ func (s *Server) handlePackApply(w http.ResponseWriter, r *http.Request) {
"skipped": result.Skipped,
"paused": result.Paused,
"resumed": result.Resumed,
// Exact before/after values let the dashboard warn before a restart
// activates a different evaluation cadence.
"governor_changes": result.GovernorChanges,
// Empty unless the operator deleted one of this level's agents.
"tombstoned": result.Tombstoned,
})
Expand Down Expand Up @@ -507,12 +567,13 @@ func (s *Server) handlePackSetLevel(w http.ResponseWriter, r *http.Request) {
s.auditFromRequest(r, "set_acmm_level", auditDetail("level", strconv.Itoa(body.Level)), "")
s.logger.Info("ACMM level set", "level", body.Level, "paused", len(paused), "resumed", len(resumed), "packUpdated", packUpdated)
jsonResponse(w, map[string]interface{}{
"ok": true,
"level": body.Level,
"packAgents": packAgentNames,
"packUpdated": packUpdated,
"paused": paused,
"resumed": resumed,
"ok": true,
"level": body.Level,
"packAgents": packAgentNames,
"packUpdated": packUpdated,
"governor_changes": packResult.GovernorChanges,
"paused": paused,
"resumed": resumed,
})
}

Expand Down
89 changes: 89 additions & 0 deletions src/pkg/dashboard/api_packs_governor_changes_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
package dashboard

import (
"bytes"
"encoding/json"
"log/slog"
"net/http/httptest"
"strings"
"testing"

"github.com/kubestellar/hive/pkg/config"
)

func TestApplyPackForceReportsGovernorChanges(t *testing.T) {
srv := newFullServer(t)
var logs bytes.Buffer
logger := slog.New(slog.NewTextHandler(&logs, &slog.HandlerOptions{Level: slog.LevelWarn}))
srv.logger = logger
srv.deps.Logger = logger

pack, err := config.ACMMPackByLevel(5)
if err != nil {
t.Fatal(err)
}
wantInterval := pack.Governor.EvalIntervalS
wantCadence := pack.Governor.Cadences["surge"]["scanner"]
if wantInterval == 0 || wantCadence == "" {
t.Skip("L5 pack no longer has the governor settings this regression covers")
}

srv.deps.Config.Governor.EvalIntervalS = 3600
srv.deps.Config.Governor.Modes = map[string]config.ModeConfig{
"surge": {Cadences: map[string]config.Cadence{"scanner": "pause"}},
}

result, err := srv.ApplyPackForce(5)
if err != nil {
t.Fatalf("ApplyPackForce(5): %v", err)
}
if result.GovernorChanges == nil || result.GovernorChanges.EvalIntervalS == nil {
t.Fatal("governor interval change was not reported")
}
if got := result.GovernorChanges.EvalIntervalS; got.From != 3600 || got.To != wantInterval {
t.Errorf("interval change = %+v, want 3600 -> %d", got, wantInterval)
}

foundCadence := false
for _, change := range result.GovernorChanges.Cadences {
if change.Mode == "surge" && change.Agent == "scanner" {
foundCadence = true
if change.From != "pause" || change.To != wantCadence {
t.Errorf("scanner surge change = %+v, want pause -> %q", change, wantCadence)
}
}
}
if !foundCadence {
t.Errorf("scanner surge cadence change missing from %+v", result.GovernorChanges.Cadences)
}
if got := logs.String(); !strings.Contains(got, "ACMM pack changed governor settings") || !strings.Contains(got, "governor_eval_interval_s_from=3600") || !strings.Contains(got, "governor_eval_interval_s_to=") || !strings.Contains(got, "cadence_changes") {
t.Errorf("WARN log does not identify governor changes: %s", got)
}
}

func TestPackApplyResponseIncludesGovernorChanges(t *testing.T) {
srv := newFullServer(t)
srv.deps.Config.Governor.EvalIntervalS = 3600

req := httptest.NewRequest("POST", "/api/packs/5/apply", nil)
req.SetPathValue("level", "5")
w := httptest.NewRecorder()
markOwnerRequest(req)
srv.handlePackApply(w, req)

if w.Code != 200 {
t.Fatalf("status = %d, body=%s", w.Code, w.Body.String())
}
var response struct {
GovernorChanges *GovernorChanges `json:"governor_changes"`
}
if err := json.NewDecoder(w.Body).Decode(&response); err != nil {
t.Fatal(err)
}
if response.GovernorChanges == nil || response.GovernorChanges.EvalIntervalS == nil {
t.Fatalf("response omits evaluation interval change: %+v", response.GovernorChanges)
}
if got := response.GovernorChanges.EvalIntervalS.From; got != 3600 {
t.Errorf("response interval from = %d, want 3600", got)
}
}
23 changes: 21 additions & 2 deletions src/pkg/dashboard/static/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -15532,6 +15532,7 @@ <h3 style="color:var(--amber);text-align:center">Configure LiteLLM</h3>
}
const created = (data.created || []).length;
const skipped = (data.skipped || []).length;
const governorChanges = formatGovernorChanges(data.governor_changes);
closeACMMDialog();
window._acmmOverride = { level: level, packAgents: data.packAgents || [] };
if (window._lastStatus) {
Expand All @@ -15540,7 +15541,8 @@ <h3 style="color:var(--amber);text-align:center">Configure LiteLLM</h3>
window._lastStatus.acmmPackAgents = data.packAgents || [];
render(window._lastStatus);
}
showToast(`Level ${level} applied: ${created} created, ${skipped} skipped`, 'success');
showToast(`Level ${level} applied: ${created} created, ${skipped} skipped` + (governorChanges ? ' — governor settings changed' : ''), governorChanges ? 'info' : 'success');
if (governorChanges) await hiveAlert(governorChanges, 'Governor settings changed');
_acmmPacks = null;
} catch (err) {
hideACMMLoading();
Expand All @@ -15567,6 +15569,7 @@ <h3 style="color:var(--amber);text-align:center">Configure LiteLLM</h3>
errEl.style.display = 'block';
return;
}
const governorChanges = formatGovernorChanges(data.governor_changes);
closeACMMDialog();
window._acmmOverride = { level: level, packAgents: data.packAgents || [] };
if (window._lastStatus) {
Expand All @@ -15575,7 +15578,8 @@ <h3 style="color:var(--amber);text-align:center">Configure LiteLLM</h3>
window._lastStatus.acmmPackAgents = data.packAgents || [];
render(window._lastStatus);
}
showToast(`ACMM level set to L${level} (${name})`, 'success');
showToast(`ACMM level set to L${level} (${name})` + (governorChanges ? ' — governor settings changed' : ''), governorChanges ? 'info' : 'success');
if (governorChanges) await hiveAlert(governorChanges, 'Governor settings changed');
_acmmPacks = null;
} catch (err) {
hideACMMLoading();
Expand All @@ -15584,6 +15588,21 @@ <h3 style="color:var(--amber);text-align:center">Configure LiteLLM</h3>
}
}

// Packs reconcile scheduling as part of an explicit level apply. Show the
// exact mutation instead of leaving an operator to discover it after restart.
function formatGovernorChanges(changes) {
if (!changes) return '';
const lines = [];
const interval = changes.eval_interval_s;
if (interval && interval.from !== interval.to) {
lines.push(`Evaluation interval: ${interval.from}s → ${interval.to}s`);
}
for (const cadence of changes.cadences || []) {
lines.push(`${cadence.mode} / ${cadence.agent}: ${cadence.from || 'unset'} → ${cadence.to || 'unset'}`);
}
return lines.join('\n');
}

// Short "L3 Quality-Gated"-style label for an ACMM pack level. Shared by the
// badges, the level dialog subtitle, and the Governor config General tab.
function acmmLevelLabel(level) {
Expand Down
Loading