artifact: refresh manifest at runtime without restart (atomic snapshot + auto-refresh)

[mushrafmim]

Problem

The artifact registry reads the manifest (manifest.json) exactly once, at startup: LoadManifest → RegisterFromConfig populates the registry's in-memory manifest table, which is then treated as read-only for the lifetime of the process (see artifact/registry.go, artifact/manifest.go).

As a result, any change to the manifest table — adding a new artifact row, or changing a version/path — is invisible until the process is restarted.

Note this is narrower than "the manifest isn't synced":

• Artifact content is already live. loadAndParse calls loader.Load(...) on every Get/Latest, so editing the bytes of an already-registered artifact is picked up without a restart.
• The manifest table (id/kind/version → path) is frozen at startup. Only new/changed rows require a restart.

For backends where artifacts are expected to evolve independently of binary deploys (e.g. a repo- or bucket-backed registry), requiring a restart to pick up a new manifest row undercuts the point of an external, versioned registry — the artifact lifecycle should be decoupled from the deployment lifecycle.

Proposed Solution

Make the manifest table refreshable at runtime, without blocking readers and without pushing any new responsibility onto individual loaders (the refresh policy lives in the registry, above the Loader interface).

1. Atomic snapshot swap (foundation). Move the manifest table behind an atomic.Pointer[snapshot] (sync/atomic, available on the repo's Go version). Readers on the hot path (Get/Latest) do a single lock-free atomic load. A snapshot is immutable once published.

2. Reload(ctx) error. Re-fetch manifest.json through the existing loader, build a fresh table, validate every row, and only then atomically swap it in. On any error, return it and keep the last-good snapshot — a transient fetch/parse failure must never blank the catalog.

3. StartAutoRefresh(ctx, interval). A registry-owned background poller that calls Reload every interval, stopping when ctx is cancelled. interval <= 0 is a no-op (feature off), so it is strictly opt-in. The interval comes from the application's config and is passed in here — it is a registry concern, not a loader concern, so loaders stay unaware of refresh entirely.

Failures during polling are logged (slog) and leave the current snapshot in place.

Why an atomic snapshot instead of a mutex: this is a read-heavy / write-rare workload (many Gets between infrequent refreshes) where the whole table is replaced as a unit and slight staleness is acceptable — the textbook case where a copy-on-write snapshot swap avoids taxing every read with lock overhead.

Out of scope (separate follow-ups):

• Artifact content caching (versioned paths are immutable and cacheable indefinitely; Latest/unversioned entries want a TTL).
• Any specific trigger wiring beyond the poller (signal handler, admin endpoint, webhook) — the app can drive Reload directly if it wants those.

Alternatives

• RWMutex guarding the table. Correct, but blocks readers during the swap and taxes every Get with read-lock overhead to guard against a write that almost never happens. Rejected in favor of the lock-free snapshot.
• Explicit Reload() only, no poller. Smallest surface, but pushes the "when to refresh" burden onto every consumer. Keeping Reload() public still allows this, so the poller is additive rather than exclusive.
• Restart / rolling redeploy to pick up manifest changes. The status quo. Acceptable for file-mounted deployments where a manifest change already coincides with a redeploy, but defeats the decoupling for registries backed by an independently-updated source.
• Per-loader refresh. Would duplicate the same polling logic across every backend and leak a registry-level policy into the Loader interface. Rejected — refresh belongs in the registry.