feat: persistent hilt openbao vault

[alanshaw]

Makes the Hilt vault persistent so it survives make down / make up.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved snapshot compatibility, embedding coverage, and persistence test issues remain.

Review effort: Lite
Findings: 1 · 1

Open (2)

• Update snapshots or handle missing Hilt vault volumes · New
• Add Hilt OpenBao paths to embedded file existence test · New

What changed in this PR

Makes Hilt’s OpenBao vault persistent across restarts and snapshots.

Changes:

• Adds raft-backed Hilt vault storage and initialization.
• Shares OpenBao bootstrap logic with Ingot.
• Updates snapshot volumes, embedding, tests, scripts, and documentation.

File	Summary	Findings
systems/​ingot/​README.md	Documents shared bootstrap and vault behavior.	Nit (1 vote): Update stale guidance claiming Hilt vault is in-memory and requires rerunning s3-key after restarts.
systems/​ingot/​openbao/​init.sh	Uses shared bootstrap logic.	No final comment.
systems/​ingot/​compose.yml	Mounts shared bootstrap resources.	No final comment.
systems/​hilt/​README.md	Documents persistent vault lifecycle.	No final comment.
systems/​hilt/​openbao/​init.sh	Initializes and provisions the Hilt vault.	No final comment.
systems/​hilt/​openbao/​config.hcl	Configures raft storage.	No final comment.
systems/​hilt/​compose.yml	Adds persistent vault and initializer services.	Moderate (1 vote): Add an end-to-end regression verifying keys survive restart or snapshot restore, including raft restore, unseal, and token provisioning.
systems/​common/​openbao/​bootstrap.sh	Provides shared initialization and unseal logic.	No final comment.
scripts/​s3-key.sh	Updates persistence-related behavior.	Nit (1 vote): Update stale guidance in systems/ingot/README.md and docs/PERF_TESTING.md, while retaining the pre-persistence snapshot caveat.
README.md	Updates the Hilt vault description.	No final comment.
pkg/​snapshot/​volumes.go	Includes Hilt vault volumes in snapshots.	Critical (1 vote): Update the committed snapshot or add compatibility handling; older snapshots restore tenants without their vault keys.
images_test.go	Covers the initializer image.	No final comment.
embed.go	Embeds OpenBao assets.	Moderate (2 votes): Add both Hilt OpenBao paths to TestEmbeddedFilesExist.
docs/​SNAPSHOTS.md	Documents new snapshot archives.	No final comment.
CLAUDE.md	Updates development guidance.	No final comment.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[bajtos]

Nice!

I assumed it was an intentional design decision to use ephemeral OpenBao with in-memory storage only. I am wondering about the ramifications of this change. I guess we can address them as we discover them along the way. 🤷🏻

[alanshaw]

I assumed it was an intentional design decision to use ephemeral OpenBao with in-memory storage only.

No just short sightedness I think!