release: chart 0.12.3 (Dakera server v0.12.3, dashboard 0.5.2) + hardening

.github/workflows/helm-ci.yml

@@ -23,10 +23,10 @@ jobs:
     name: Lint and render the chart
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262  # v4.4.0
 
       - name: Set up Helm
-        uses: azure/setup-helm@v4
+        uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4  # v4.3.1
         with:
           version: v3.14.4
 

.github/workflows/helm-publish-oci.yml

@@ -18,17 +18,27 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262  # v4.4.0
 
       - name: Set up Helm
-        uses: azure/setup-helm@v4
+        uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4  # v4.3.1
         with:
           version: v3.14.4
 
       - name: Log in to GHCR
         run: echo "${{ secrets.GITHUB_TOKEN }}" | helm registry login ghcr.io -u "${{ github.repository_owner }}" --password-stdin
 
-      - name: Package + push chart
+      # Every change under charts/ reaches this job, but a published version is never
+      # overwritten: OCI tags are mutable, and users who pinned --version X must keep
+      # getting what X was. A change is published with the next version bump
+      # (charts/dakera/Chart.yaml); chart-releaser skips existing versions the same way
+      # (cr.yaml skip-existing).
+      - name: Package + push chart (new versions only)
         run: |
+          VERSION=$(helm show chart charts/dakera | awk '/^version:/ {print $2}')
+          if helm show chart oci://ghcr.io/dakera-ai/dakera-helm/dakera --version "$VERSION" >/dev/null 2>&1; then
+            echo "::notice::dakera $VERSION is already published to oci://ghcr.io/dakera-ai/dakera-helm; not overwriting it (bump the chart version to publish)"
+            exit 0
+          fi
           helm package charts/dakera
           helm push dakera-*.tgz oci://ghcr.io/dakera-ai/dakera-helm

README.md

@@ -18,10 +18,10 @@ Deploy Dakera on Kubernetes in seconds. Two install paths: OCI registry (no repo
 
 | | Dakera server | Chart version | Where |
 |---|---|---|---|
-| **Latest** | **v0.12.2** | **0.12.2** (and later 0.12.x) | `main` (this branch), the Helm repository and the OCI registry |
+| **Latest** | **v0.12.3** | **0.12.3** (and later 0.12.x) | `main` (this branch), the Helm repository and the OCI registry |
 | Previous | v0.11.x (last: v0.11.108) | 0.11.x | the [`release/0.11`](https://github.com/Dakera-AI/dakera-helm/tree/release/0.11) branch |
 
-`main` and the latest chart are **v0.12.2**. The v0.11 chart is kept on the
+`main` and the latest chart are **v0.12.3**. The v0.11 chart is kept on the
 `release/0.11` branch, unchanged, so a v0.11 setup stays findable and working.
 To stay on v0.11, pin the old chart version, for example:
 
@@ -35,7 +35,7 @@ helm install dakera dakera/dakera --version 0.11.107 ...
 set `--set dakera.image.tag=0.11.108` to run the final v0.11 server.
 `helm search repo dakera/dakera --versions` lists every published version. The
 chart version always equals the Dakera server version it deploys. Upgrading from
-v0.11, v0.12.0 or v0.12.1 to v0.12.2: read [charts/dakera/CHANGELOG.md](charts/dakera/CHANGELOG.md)
+v0.11 or v0.12.0–v0.12.2 to v0.12.3: read [charts/dakera/CHANGELOG.md](charts/dakera/CHANGELOG.md)
 and the server's
 [UPGRADE.md](https://github.com/Dakera-AI/dakera/blob/main/docs/v0.12/UPGRADE.md).
 
@@ -56,7 +56,7 @@ Pin a specific version:
 
 ```bash
 helm install dakera oci://ghcr.io/dakera-ai/dakera-helm/dakera \
-  --version 0.12.2 \
+  --version 0.12.3 \
   --namespace dakera --create-namespace \
   --set dakera.rootApiKey=<your-key> \
   --set minio.rootPassword=<your-password>
@@ -84,6 +84,32 @@ helm upgrade dakera dakera/dakera --namespace dakera \
   --set minio.rootPassword=<your-password>
 ```
 
+### Upgrading from chart 0.12.2
+
+Chart 0.12.3 deploys server 0.12.3 and Dashboard 0.5.2; nothing is migrated by hand. Back up first
+(`POST /admin/backups`), then `helm upgrade` (from 0.12.0 / 0.12.1, the notes below apply too). What changes:
+
+- **Server image:** distroless, uid/gid 1000, no `curl` or shell tools (the probes are `httpGet`). The GPU image is
+  built on CUDA 13: GPU nodes need NVIDIA driver **R580 or newer**.
+- **API:** `POST /ops/shutdown` needs a global `super_admin` key (nothing in the chart calls it);
+  `POST /admin/keys`, `/v1/namespaces/{ns}/keys`, `/admin/backups` and `/admin/encryption/rotate-key` refuse
+  unknown JSON fields (422). Recall gains `neighborhood_mode` / `include_summaries` (server CHANGELOG 0.12.3).
+- **Connections:** an idle keep-alive (and HTTP/2) connection is closed after
+  `DAKERA_HTTP_HEADER_READ_TIMEOUT_SECS` (30 s): see "Ingress and load balancers". The shutdown drains REST
+  connections for `shutdownDrainSecs` (8 s); `dakera.terminationGracePeriodSeconds` (30) must hold 15 + it + 5.
+- **Authentication off** (`dakera.config.authEnabled: false`): a request whose `Host` is a public dotted name
+  (the ingress host) gets **421** unless listed in `dakera.config.allowedHosts`. `http://<fullname>:3000`, IPs and
+  private-use names (`*.local`, `*.internal`, `*.home.arpa`: the Service FQDN
+  `<fullname>.<namespace>.svc.cluster.local`) keep working.
+- **Autopilot deduplication is off by default** (earlier releases merged near-duplicates every 6 hours): add
+  `DAKERA_AUTOPILOT_DEDUP_MODE=merge` to `dakera.extraEnv` to keep it. A full queue answers `429` + `Retry-After`
+  (was `503`); recall of an agent that has stored nothing answers `200` with no memories. The server raises its
+  open-files limit to the hard limit at startup, sizes its model pools to the physical cores, and, under a memory
+  limit with room for them, prepacks the model weights (faster, more memory; values.yaml, `extraEnv`).
+- **Dashboard 0.5.2** runs as uid/gid 10001 with a read-only root (the chart mounts `/tmp`); new optional
+  `dashboard.trustedProxies` (`DAKERA_TRUSTED_PROXIES`). Chart defaults that changed: read-only root filesystem for
+  the server, the MinIO NetworkPolicy, no service-account tokens (CHANGELOG.md, 0.12.3, upgrade notes).
+
 ### Upgrading from chart 0.12.1
 
 Chart 0.12.2 deploys server 0.12.2 and Dashboard 0.5.1; nothing is migrated by hand. Back up first
@@ -103,7 +129,7 @@ Chart 0.12.2 deploys server 0.12.2 and Dashboard 0.5.1; nothing is migrated by h
   `mem_s` + 24 hex ids, memory-shaped writes through the raw vector routes of an agent namespace, agent ids
   over 241 bytes, content over `DAKERA_MAX_MEMORY_CONTENT_BYTES` (100000 bytes) on update as well as store.
 - **Storage:** the RocksDB hot tier writes binary records. Back to chart 0.12.1 / 0.12.0: nothing to do. Back
-  to v0.11.108: the rollback Job with the 0.12.2 image rewrites them as JSON (see "Rolling back to v0.11").
+  to v0.11.108: the rollback Job with the 0.12.2 (or later) image rewrites them as JSON (see "Rolling back to v0.11").
 
 New optional values (rendered only when set; empty = the server's default): `dakera.config.sessionIdleTimeoutSecs`,
 `maxMemoryContentBytes`, `vectorCacheBytes`, `rocksdbRecordFormat` (see "Common Options" and values.yaml). Details:
@@ -155,8 +181,8 @@ Pass values with `--set key=value` or a `values.yaml` file (`-f values.yaml`).
 
 | Key | Description |
 |---|---|
-| `dakera.rootApiKey` | Root API key for Dakera. **Required.** |
-| `minio.rootPassword` | MinIO root password for S3-compatible object storage. **Required.** |
+| `dakera.rootApiKey` | Root API key for Dakera. **Required**, unless `dakera.existingSecret` names a Secret that holds it. |
+| `minio.rootPassword` | MinIO root password for S3-compatible object storage. **Required** with `dakera.rootApiKey`; with `dakera.existingSecret`, that Secret's `MINIO_ROOT_PASSWORD`. |
 
 ### Common Options
 
@@ -187,6 +213,16 @@ Pass values with `--set key=value` or a `values.yaml` file (`-f values.yaml`).
 | `ingress.enabled` | `false` | Enable ingress |
 | `mcp.enabled` | `false` | dakera-mcp speaks MCP over stdio only: as a pod it exits at once (CrashLoopBackOff, measured with 0.10.11). Run it next to the MCP client |
 | `ingress.apiHost` / `ingress.dashboardHost` / `ingress.mcpHost` | unset | Ingress hostnames |
+| `dakera.existingSecret` | `""` | A Secret you manage instead of `<release>-secrets` |
+| `dakera.config.documents.*` | `""` | Document import (server 0.12.3, on by default). Extraction uses up to 2 x 1024 MB on top of the server: size `dakera.resources.limits.memory` (4Gi) for it |
+| `ingress.annotations` | `proxy-body-size: 500m` | ingress-nginx's own default (1m) refuses document uploads (25 MiB) and backups |
+| `dakera.readOnlyRootFilesystem` | `true` | Read-only root filesystem (the data root, the model cache and `/tmp` are volumes) |
+| `minio.networkPolicy.enabled` | `true` | Only this release's Dakera pods reach the built-in MinIO |
+| `dakera.config.httpHeaderReadTimeoutSecs`, `maxConnections`, `shutdownDrainSecs`, `allowedHosts`, `neighborhoodMode`, ... | `""` | Server 0.12.3 settings, rendered only when set (chart README). With `authEnabled: false`, `allowedHosts` must list the dotted names clients use (else 421) |
+
+Behind ingress-nginx or a load balancer, read the chart README, "Ingress and load balancers": from the server
+release 0.12.3 an idle keep-alive connection is closed after `DAKERA_HTTP_HEADER_READ_TIMEOUT_SECS` (30 s),
+so the proxy's upstream idle timeout must be shorter. Security defaults: chart README, "Security defaults".
 
 Full values reference: [charts/dakera/values.yaml](charts/dakera/values.yaml)
 
@@ -215,7 +251,7 @@ variables, constraints, sizing and verification per feature, is
 
 ```bash
 # multilingual search on a fresh install (from a clone of this repository, for the example file)
-helm install dakera oci://ghcr.io/dakera-ai/dakera-helm/dakera --version 0.12.2 -n dakera --create-namespace \
+helm install dakera oci://ghcr.io/dakera-ai/dakera-helm/dakera --version 0.12.3 -n dakera --create-namespace \
   --set dakera.rootApiKey=$(openssl rand -hex 32) --set minio.rootPassword=$(openssl rand -hex 16) \
   -f charts/dakera/examples/values-multilingual.yaml
 # several combinable features: examples/values-combined.yaml (multilingual + multimodal + records + rabitq)
@@ -395,7 +431,7 @@ Ready 25 s after install.
 - Pre-pull before the pod serves: `dakera.models.pull: [configured]` (or a list of names).
 - Keep the cache across pod restarts: `dakera.models.persistence.enabled: true`.
 - Air-gapped: seed that volume from a host that can reach the Hub
-  (`docker run --rm -v <volume>:/app/models ghcr.io/dakera-ai/dakera:0.12.2 models pull --dir /app/models <model>`,
+  (`docker run --rm -v <volume>:/app/models ghcr.io/dakera-ai/dakera:0.12.3 models pull --dir /app/models <model>`,
   then mount it as the PVC), and use only models the image or the volume holds.
   See the server's [models-and-docker.md](https://github.com/Dakera-AI/dakera/blob/main/docs/models-and-docker.md).
 
@@ -470,8 +506,8 @@ helm upgrade dakera oci://ghcr.io/dakera-ai/dakera-helm/dakera --version 0.11.10
   --set dakera.image.tag=0.11.108 <your v0.11 values>
 ```
 
-Run the Job with the **0.12.2 image** (the default: the server's image; leave `rollback.image` empty).
-0.12.2 writes binary RocksDB hot-tier records, and its `dakera downgrade` rewrites them as JSON before
+Run the Job with the **server's image, 0.12.3** (the default; leave `rollback.image` empty).
+0.12.2 and later write binary RocksDB hot-tier records, and their `dakera downgrade` rewrites them as JSON before
 v0.11.108 can read them (report field `hot_tier_records_rewritten_as_json`); an older v0.12 image would not.
 v0.11.108 cannot address namespace names or record ids longer than 200 bytes that 0.12.2 stored (server
 [rollback.md](https://github.com/Dakera-AI/dakera/blob/main/docs/v0.12/rollback.md), row 23b). Going back

charts/dakera/CHANGELOG.md

@@ -3,6 +3,79 @@
 The chart version equals the Dakera server version it deploys. v0.11 lives on the
 `release/0.11` branch and in the 0.11.x chart versions (last published: 0.11.107).
 
+## 0.12.3 (Dakera server v0.12.3)
+
+Deploys server **0.12.3** (`ghcr.io/dakera-ai/dakera:0.12.3`) and Dashboard **0.5.2**
+(`ghcr.io/dakera-ai/dakera-dashboard:0.5.2`). Released after both images are published.
+
+### Upgrade notes (0.12.0 – 0.12.2 to 0.12.3)
+
+Nothing is migrated by hand: back up (`POST /admin/backups`), then `helm upgrade`. Server behaviour that changes:
+`POST /ops/shutdown` needs a global `super_admin` key; four admin routes refuse unknown JSON fields (422); the image
+is distroless (uid/gid 1000, no `curl` or shell tools; the probes are `httpGet`); the GPU image is CUDA 13 (NVIDIA
+driver R580+). With **authentication off**, a request whose `Host` is a public dotted name (the ingress host) gets
+421 unless listed in `dakera.config.allowedHosts` (the Service FQDN, a `*.local` name, is accepted; NOTES prints the
+names). Autopilot deduplication is off by default (`DAKERA_AUTOPILOT_DEDUP_MODE=merge` in `dakera.extraEnv` keeps
+it); a full queue answers `429` + `Retry-After` (was `503`). Dashboard 0.5.2 runs as
+uid/gid 10001 with a read-only root and a `/tmp` emptyDir (`dashboard.securityContext`; an image up to 0.5.1 needs
+the old root settings, see values.yaml). Chart defaults that changed:
+
+1. **The server's root filesystem is read-only by default** (`dakera.readOnlyRootFilesystem: true`, was `false`),
+   also for the models init container and the rollback Job. Everything the server writes is on a volume: the data
+   root, the model cache and `/tmp`. With `dakera.persistence.enabled: false` the data root is now an `emptyDir`
+   (before: the container's own filesystem; both are lost with the pod). Measured with server 0.12.2 (store,
+   recall, backups). `--set dakera.readOnlyRootFilesystem=false` restores the previous behaviour.
+2. **The built-in MinIO accepts connections only from this release's Dakera pods** (new
+   `minio.networkPolicy.enabled: true`), where the network plugin enforces NetworkPolicy. Anything else that used
+   the bucket or scraped MinIO (a Prometheus) goes in `minio.networkPolicy.additionalFrom`, or set
+   `minio.networkPolicy.enabled=false`. `kubectl port-forward` is not affected.
+3. **Behind ingress-nginx or a load balancer**: the server closes an idle keep-alive (or HTTP/2)
+   connection after `DAKERA_HTTP_HEADER_READ_TIMEOUT_SECS` (30 s). Keep the proxy's upstream idle timeout below it
+   (ingress-nginx `upstream-keepalive-timeout`, 60 s by default) or raise `dakera.config.httpHeaderReadTimeoutSecs`
+   above it; README, "Ingress and load balancers".
+
+### Added
+
+- Optional `dakera.config` settings of server 0.12.3, rendered only when set (an image up to 0.12.2 would list
+  them as unknown) and checked by `values.schema.json`: `httpHeaderReadTimeoutSecs`
+  (`DAKERA_HTTP_HEADER_READ_TIMEOUT_SECS`, 1 to 86400), `neighborhoodMode` (`inline` / `separate` / `off`),
+  `neighborhoodEventTime`, `recallRankDepth` (1 to 1000), `recallCollapseDerived`, `recallExcludeSummaries`,
+  `recallSessionCap`, `sentenceDecomp`, `supersedesEventTime`.
+- `dakera.existingSecret`: a Secret you manage (`DAKERA_ROOT_API_KEY`, and with the built-in MinIO
+  `MINIO_ROOT_PASSWORD`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) instead of `<release>-secrets`; no secret in
+  the values. Refused together with `dakera.rootApiKey`.
+- `minio.networkPolicy` (see above); `dakera.probes.timeoutSeconds` (5); `dashboard.securityContext`.
+- Server 0.12.3 settings `maxConnections` (`DAKERA_MAX_CONNECTIONS`), `shutdownDrainSecs`
+  (`DAKERA_SHUTDOWN_DRAIN_SECS`) and `allowedHosts` (`DAKERA_ALLOWED_HOSTS`), rendered only when set; NOTES warns
+  when authentication is off. New `dakera.terminationGracePeriodSeconds` (30): the chart refuses a
+  `shutdownDrainSecs` with which 15 s of media jobs + the drain + 5 s would not fit.
+- `dakera.config.documents.*`: the document-import settings of server 0.12.3 (`DAKERA_DOCUMENTS`, `DAKERA_DOCUMENT_*`),
+  each rendered only when set and range-checked. Document import is on by default in the server: PDF / DOCX
+  extraction runs in child processes using up to `extractConcurrency` x `extractMemoryMb` (2 x 1024 MB) inside the
+  pod's memory limit (4Gi by default): raise it or lower the concurrency for PDF-heavy imports.
+- `ingress.annotations` now defaults to `nginx.ingress.kubernetes.io/proxy-body-size: "500m"`: ingress-nginx's own
+  1m limit refused document uploads (25 MiB), imports and backups. Your annotations are merged with it.
+- `dashboard.trustedProxies` (`DAKERA_TRUSTED_PROXIES`, dashboard 0.5.2): the proxies whose `X-Forwarded-For` names
+  the client, for the sign-in throttle.
+
+### Changed
+
+- Every pod: `automountServiceAccountToken: false` (nothing in the chart calls the Kubernetes API) and the
+  `RuntimeDefault` seccomp profile. Server probes: `timeoutSeconds: 5` (the 1 s default fails a busy server); they
+  stay `httpGet`, which needs nothing in the image (newer server images have no `curl`).
+- Dashboard 0.5.1 -> 0.5.2: runs as uid/gid 10001, read-only root filesystem with a `/tmp` emptyDir, no
+  capability, no privilege escalation (checked on the 0.5.2 runtime), and `global.imagePullSecrets` like the other
+  pods. Its probes stay `httpGet` (the image has no `curl`). MCP: no privilege escalation, no capability.
+- The "create the Secret yourself" instructions printed without `rootApiKey` used `minioadmin` as the S3 access key
+  while the built-in MinIO runs as `minio.rootUser`: they now use `minio.rootUser`.
+- No `/ops/shutdown` caller in the chart: the server's new `super_admin` requirement for it needs no change here.
+- Checked against the final server 0.12.3: every `DAKERA_*` name the chart renders is one it accepts (`--check-config`
+  on the default render and with every 0.12.3 setting set). With authentication off, the Service FQDN (a `*.local`
+  name) needs no `allowedHosts` entry: the server accepts private-use names (`*.local`, `*.internal`,
+  `*.home.arpa`); NOTES, values.yaml and the README now say so and only ask for public names (the ingress host).
+- `scripts/validate-chart.sh`: `helm template ... | grep -q` under `pipefail` could fail with SIGPIPE (141) when
+  helm was still writing; the checks read their whole input now.
+
 ## 0.12.2 (Dakera server v0.12.2)
 
 Deploys server **0.12.2** (`ghcr.io/dakera-ai/dakera:0.12.2`) and Dashboard