refactor(rpc): split the policy write out of the management PUT for the bucket service

AGENTS.md

@@ -2,10 +2,23 @@
 
 Tenant-management service for the Forge network. Hilt owns tenants, their access
 keys, and their buckets — plus the UCAN delegations and key material that back
-them. It exposes two APIs and talks to one external service:
+them. It exposes two APIs and talks to three external services:
 
-- **Tenant REST API** (`pkg/api`, echo) — partner-facing CRUD for tenants and
-  access keys, guarded by a pre-shared partner key.
+- **Tenant REST API** (`pkg/api`, echo) — partner-facing CRUD for tenants,
+  access keys, principals and bucket policies, guarded by a pre-shared partner
+  key. `POST /tenants/{id}/access-keys` creates both key kinds. Without
+  `principalId` it is a **service key**: it carries its own `permissions` and
+  `buckets`, and the tenant→access-key delegations for them are issued at
+  creation. With `principalId` it is a **principal-bound key**: it takes no
+  permissions or buckets (both stored as `NULL`) and names a principal (the
+  console's `principalId`). What that principal may do on a bucket is the
+  bucket policy's effective set for it, and the key holds, over each bucket
+  the principal can reach, the tenant→access-key delegations for the Forge
+  commands that set maps to (`pkg/grant`), rewritten on every policy change.
+  Each authorize evaluates the policy and re-delegates from the key to the
+  gateway, as a service key does. Removing a principal tombstones its row
+  (`deleted_at`): no read returns it, and a later `PUT` for the same id
+  revives it with no keys and named in no statement.
 - **Hilt UCAN RPC API** (`pkg/rpc`, ucantone server mounted at `POST /`) — the
   `/s3/*` commands Ingot (the S3 gateway) invokes: `/s3/request/authorize`,
   `/s3/bucket/{create,delete,info,list}`; and the self-issued admin commands
@@ -14,6 +27,14 @@ them. It exposes two APIs and talks to one external service:
   bucket's storage space and to manage routing policies (`pkg/client`): each
   provider owns a policy whose candidates are its storage nodes, and every
   bucket's space is pointed at its provider's policy on creation.
+- **Swarf** (the revocation service) — Hilt calls its `/ucan/revoke` to
+  revoke delegations, every revocation of one write in a single request
+  (`PublishBatch`): a deleted key's grants, a deleted bucket's, and the grants
+  a policy change or a principal removal takes off a principal-bound key
+  (`pkg/grant`). No revocation carries a nonce, so a retry is a duplicate
+  Swarf records once.
+- **PLC** (the did:plc directory) — Hilt creates a tenant's did:plc there on
+  provisioning and deactivates it on deletion (`pkg/fx/plc.go`).
 
 Module: `github.com/fil-forge/hilt` (Go 1.27). Sibling repos it builds on:
 `ucantone` (UCAN primitives: `did`, `multikey`, `ucan/delegation`, `binding`,
@@ -41,7 +62,16 @@ and `sprue` (the upload service; mirror its patterns where relevant).
   images Docker never re-pulls — `docker pull` them when the stack misbehaves,
   or override per run with `HILT_ITEST_UPLOAD_IMAGE` / `HILT_ITEST_PIRI_IMAGE`
   / `HILT_ITEST_INGOT_IMAGE` / `HILT_ITEST_SWARF_IMAGE` / `HILT_ITEST_PIRI_BINARY`
-  / `HILT_ITEST_SWARF_BINARY`. CI runs the suite on
+  / `HILT_ITEST_SWARF_BINARY` / `HILT_ITEST_INGOT_BINARY`. A binary override
+  wants a static linux build for the Docker host's architecture
+  (`GOOS=linux GOARCH=<host arch> CGO_ENABLED=0 GOWORK=off go build`), and is
+  how the IAM scenarios run against ingot changes that the `:main` image does
+  not carry yet. Until the published `:main` ingot image carries the IAM
+  changes, the IAM scenarios skip unless `HILT_ITEST_INGOT_BINARY` is set (or
+  `HILT_ITEST_IAM=1`); drop that guard once it does. CI runs the suite after
+  the unit job, on pull requests, on pushes to `main`, and on manual dispatch
+  (`.github/workflows/go-test.yml`). It sets neither override, so the IAM
+  scenarios stay skipped there until the published image carries the change.
 - Editor/LSP diagnostics can lag after cross-file or cross-package edits —
   `go build` / `go vet` are authoritative, prefer them over stale squiggles.
 
@@ -60,11 +90,20 @@ and `sprue` (the upload service; mirror its patterns where relevant).
 - `pkg/sigv4` — stdlib-only SigV4 / SigV4a verification, key derivation
   (`DeriveKey`), and local verification (`VerifyWithKey`).
 - `pkg/s3perm` — S3-permission → Forge-command mapping (shared by `api` and `rpc`).
-- `pkg/store/{tenant,accesskey,bucket,delegation,provider}` — each an interface
-  with `memory` and `postgres` backends.
+- `pkg/bucketpolicy` — the bucket policy document and its evaluation
+  (`Decode`, `Validate`, `Canonical`/`ETag`, `Effective`, `Changed`); pure, no
+  store or transport dependencies. The package doc has the document shape.
+- `pkg/grant` — an access key's stored delegations: `Issue` builds the
+  tenant→key set both key kinds hold, `Rotator` rewrites a principal-bound
+  key's set as its policies change. See the package doc.
+- `pkg/store/{tenant,accesskey,bucket,delegation,provider,principal,bucketpolicy}` —
+  each an interface with `memory` and `postgres` backends.
+- `pkg/store/pglock` — the Postgres locking helpers the stores share: bounded
+  lock waits, advisory locks, and the mapping to `store.ErrLockTimeout`.
 - `pkg/vault` (`memory`, `openbao`) — private-key storage; `paths.go` has the
   key path helpers (`TenantKeyPath`, `AccessKeyPath`).
-- `pkg/client` — clients for external services (the Sprue `UploadClient`).
+- `pkg/client` — clients for external services: the Sprue `UploadClient`, and
+  `pkg/client/management`, the partner-key REST client for the tenant API.
 - `pkg/tracing` — OpenTelemetry: `Setup` installs the OTLP exporter for
   `serve`; the Echo server-span middleware; `SpanNamer`, which names a UCAN
   request's span for its commands; `Handler`, which gives each invocation its
@@ -82,6 +121,24 @@ and `sprue` (the upload service; mirror its patterns where relevant).
   implementations kept in lockstep and exercised by one backend-parametrized test
   suite (`<entity>_test.go`). Add a method to all three (interface + both backends)
   and cover it in that suite.
+- **Locking and callbacks**: a write that another service must learn about runs
+  in one transaction: lock the row (`SELECT … FOR UPDATE`), run the
+  caller-supplied callback (`fn`, which runs inside the write before it takes
+  effect), commit. The callback's ctx carries
+  the transaction (`pglock.WithTx`), and a store that opens its own through
+  `pglock.Begin` joins it, so what the callback writes commits with the
+  caller's write or not at all. A policy write's callback rewrites the
+  affected keys' delegations and publishes the revocations before anything is
+  stored, so a failure before the commit, a failed publish included, rolls the
+  write and the rotation back together and leaves the principal with its old
+  access, never with more; a publish that succeeded followed by a commit that
+  failed leaves revoked grants without a replacement, the one window the
+  gateway's revoked set covers (`pkg/api/service/bucketpolicy` and
+  `grant.Rotator` document the sequence). A reader that
+  must not be answered from a snapshot older than an in-flight write passes
+  `store.WithShareLock()` (`SELECT … FOR SHARE`). Every lock wait is bounded by
+  `store.LockTimeout` and a statement that gives up returns
+  `store.ErrLockTimeout`, which the caller retries.
 - **RPC handlers** follow one shape: a `New<Cmd>Handler(logger, deps…) server.Route`
   constructor that returns the libforge bound command's `.Route(...)`, whose closure
   extracts `req.Invocation().Issuer()` / `req.Task().Arguments()` and delegates to an
@@ -111,10 +168,14 @@ and `sprue` (the upload service; mirror its patterns where relevant).
   and resolves every bucket it addresses within the tenant and the key's scope. A
   copy (`x-amz-copy-source` on a PUT) is two decisions: the write on the
   destination and `s3:GetObject` on the source, and the header must be a signed
-  header. Command-specific S3-permission checks stay in each handler.
-  `/s3/bucket/info` carries no signed request, so it authorizes on the
-  invocation issuer instead: the issuer must be the provider of the bucket's
-  tenant, and the access key in the arguments must belong to that same tenant.
+  header. Command-specific S3-permission checks stay in each handler. Both key
+  kinds re-delegate from the key to the gateway on `/s3/request/authorize`,
+  each per-request delegation keyed to itself. `/s3/bucket/info` carries no
+  signed request, so it authorizes on the invocation issuer instead: the issuer
+  must be the provider of the bucket's tenant, and the access key in the
+  arguments must belong to that same tenant. It answers the proof chains from
+  the bucket root through the key's stored grants over the bucket, and a
+  principal-bound key's effective actions as its permissions.
 - **Identities & keys**: tenants are secp256k1 → did:plc; access keys and buckets
   are ed25519 → did:key. Build issuers with `multikey.NewIssuer(did, signer)`. Bucket
   keys are **ephemeral** — used once to sign the bucket→tenant root delegation, then

cmd/client/accesskey/create.go

@@ -12,20 +12,28 @@ import (
 var (
 	createPermissions []string
 	createBuckets     []string
+	createPrincipal   string
 	createExpiresAt   string
 )
 
 var createCmd = &cobra.Command{
 	Use:   "create <tenant-id> <name>",
 	Short: "Create an access key for a tenant",
-	Long: "Create an access key for a tenant. The response includes the secret " +
-		"access key — this is the only time it is exposed, so capture it now.",
+	Long: "Create an access key for a tenant. Without --principal it is a service " +
+		"key carrying the permissions and buckets given here; with --principal it is " +
+		"bound to that principal and takes neither, its access coming from the " +
+		"tenant's bucket policies. The response includes the secret access key — " +
+		"this is the only time it is exposed, so capture it now.",
 	Args: cobra.ExactArgs(2),
 	RunE: func(cmd *cobra.Command, args []string) error {
+		if createPrincipal == "" && len(createPermissions) == 0 {
+			return fmt.Errorf("--permissions is required for a service key (or pass --principal)")
+		}
 		req := api.CreateAccessKeyRequest{
 			Name:        args[1],
 			Permissions: createPermissions,
 			Buckets:     createBuckets,
+			PrincipalID: createPrincipal,
 		}
 		if createExpiresAt != "" {
 			expires, err := time.Parse(time.RFC3339, createExpiresAt)
@@ -49,6 +57,6 @@ var createCmd = &cobra.Command{
 func init() {
 	createCmd.Flags().StringSliceVar(&createPermissions, "permissions", nil, "S3 permissions to grant (e.g. s3:GetObject,s3:PutObject)")
 	createCmd.Flags().StringSliceVar(&createBuckets, "buckets", nil, "bucket DIDs the key is restricted to (default: all the tenant's buckets)")
+	createCmd.Flags().StringVar(&createPrincipal, "principal", "", "bind the key to this principal (principalId); the key takes no permissions or buckets")
 	createCmd.Flags().StringVar(&createExpiresAt, "expires-at", "", "expiry as an RFC3339 timestamp (default: never expires)")
-	_ = createCmd.MarkFlagRequired("permissions")
 }

cmd/client/policy/principal.go

@@ -0,0 +1,44 @@
+package policy
+
+import (
+	"github.com/fil-forge/hilt/cmd/client/lib"
+	"github.com/spf13/cobra"
+)
+
+var listCmd = &cobra.Command{
+	Use:   "list <tenant-id> <principal-id>",
+	Short: "List the policies naming a principal",
+	Long: "List every policy of the tenant with a statement naming the principal " +
+		"or the wildcard.",
+	Args: cobra.ExactArgs(2),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		recs, err := c.ListPrincipalPolicies(cmd.Context(), args[0], args[1])
+		if err != nil {
+			return err
+		}
+		return lib.PrintJSON(cmd, recs)
+	},
+}
+
+var accessCmd = &cobra.Command{
+	Use:   "access <tenant-id> <principal-id>",
+	Short: "Show a principal's effective actions per bucket",
+	Long: "Show the actions the principal holds on each bucket, computed from the " +
+		"tenant's stored policies. Buckets it has no action on are omitted.",
+	Args: cobra.ExactArgs(2),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		buckets, err := c.GetPrincipalAccess(cmd.Context(), args[0], args[1])
+		if err != nil {
+			return err
+		}
+		return lib.PrintJSON(cmd, buckets)
+	},
+}

cmd/client/policy/root.go

@@ -0,0 +1,17 @@
+// Package policy provides the `hilt client policy` command tree: the two
+// principal reads computed from bucket policies, via the Tenant REST API, authenticated with the partner key from config
+// (auth.partner_key) or the HILT_PARTNER_KEY env var.
+package policy
+
+import "github.com/spf13/cobra"
+
+// Cmd is the `hilt client policy` command group.
+var Cmd = &cobra.Command{
+	Use:   "policy",
+	Short: "Read bucket policies by principal via the Tenant REST API",
+}
+
+func init() {
+	Cmd.AddCommand(listCmd)
+	Cmd.AddCommand(accessCmd)
+}

cmd/client/principal/access_keys.go

@@ -0,0 +1,23 @@
+package principal
+
+import (
+	"github.com/fil-forge/hilt/cmd/client/lib"
+	"github.com/spf13/cobra"
+)
+
+var listAccessKeysCmd = &cobra.Command{
+	Use:   "access-keys <tenant-id> <principal-id>",
+	Short: "List the access keys bound to a principal",
+	Args:  cobra.ExactArgs(2),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		recs, err := c.ListPrincipalAccessKeys(cmd.Context(), args[0], args[1])
+		if err != nil {
+			return err
+		}
+		return lib.PrintJSON(cmd, recs)
+	},
+}

cmd/client/principal/create.go

@@ -0,0 +1,26 @@
+package principal
+
+import (
+	"github.com/fil-forge/hilt/cmd/client/lib"
+	"github.com/spf13/cobra"
+)
+
+var createCmd = &cobra.Command{
+	Use:   "create <tenant-id> <principal-id>",
+	Short: "Record a principal of a tenant",
+	Long: "Record a principal of a tenant. A principal holds no key material and no " +
+		"delegation: its access is computed from the bucket policies naming it. " +
+		"The call is idempotent.",
+	Args: cobra.ExactArgs(2),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		rec, err := c.CreatePrincipal(cmd.Context(), args[0], args[1])
+		if err != nil {
+			return err
+		}
+		return lib.PrintJSON(cmd, rec)
+	},
+}

cmd/client/principal/delete.go

@@ -0,0 +1,27 @@
+package principal
+
+import (
+	"fmt"
+
+	"github.com/fil-forge/hilt/cmd/client/lib"
+	"github.com/spf13/cobra"
+)
+
+var deleteCmd = &cobra.Command{
+	Use:   "delete <tenant-id> <principal-id>",
+	Short: "Remove a principal from a tenant",
+	Long: "Remove a principal, its access to every bucket, and its access keys. " +
+		"The call is idempotent.",
+	Args: cobra.ExactArgs(2),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		if err := c.DeletePrincipal(cmd.Context(), args[0], args[1]); err != nil {
+			return err
+		}
+		fmt.Fprintf(cmd.OutOrStdout(), "Deleted principal %s\n", args[1])
+		return nil
+	},
+}

cmd/client/principal/get.go

@@ -0,0 +1,23 @@
+package principal
+
+import (
+	"github.com/fil-forge/hilt/cmd/client/lib"
+	"github.com/spf13/cobra"
+)
+
+var getCmd = &cobra.Command{
+	Use:   "get <tenant-id> <principal-id>",
+	Short: "Show one of a tenant's principals",
+	Args:  cobra.ExactArgs(2),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		rec, err := c.GetPrincipal(cmd.Context(), args[0], args[1])
+		if err != nil {
+			return err
+		}
+		return lib.PrintJSON(cmd, rec)
+	},
+}

cmd/client/principal/list.go

@@ -0,0 +1,23 @@
+package principal
+
+import (
+	"github.com/fil-forge/hilt/cmd/client/lib"
+	"github.com/spf13/cobra"
+)
+
+var listCmd = &cobra.Command{
+	Use:   "list <tenant-id>",
+	Short: "List a tenant's principals",
+	Args:  cobra.ExactArgs(1),
+	RunE: func(cmd *cobra.Command, args []string) error {
+		c, _, err := lib.InitManagementClient(cmd)
+		if err != nil {
+			return err
+		}
+		recs, err := c.ListPrincipals(cmd.Context(), args[0])
+		if err != nil {
+			return err
+		}
+		return lib.PrintJSON(cmd, recs)
+	},
+}

cmd/client/principal/root.go

@@ -0,0 +1,21 @@
+// Package principal provides the `hilt client principal` command tree: partner
+// operations on a tenant's principals and their access keys via the Tenant REST
+// API, authenticated with the partner key from config (auth.partner_key) or the
+// HILT_PARTNER_KEY env var.
+package principal
+
+import "github.com/spf13/cobra"
+
+// Cmd is the `hilt client principal` command group.
+var Cmd = &cobra.Command{
+	Use:   "principal",
+	Short: "Manage a tenant's principals via the Tenant REST API",
+}
+
+func init() {
+	Cmd.AddCommand(createCmd)
+	Cmd.AddCommand(listCmd)
+	Cmd.AddCommand(getCmd)
+	Cmd.AddCommand(deleteCmd)
+	Cmd.AddCommand(listAccessKeysCmd)
+}

cmd/client/root.go

@@ -5,6 +5,8 @@ package client
 import (
 	"github.com/fil-forge/hilt/cmd/client/accesskey"
 	"github.com/fil-forge/hilt/cmd/client/admin"
+	"github.com/fil-forge/hilt/cmd/client/policy"
+	"github.com/fil-forge/hilt/cmd/client/principal"
 	"github.com/fil-forge/hilt/cmd/client/tenant"
 	"github.com/spf13/cobra"
 )
@@ -20,4 +22,6 @@ func init() {
 	Cmd.AddCommand(admin.Cmd)
 	Cmd.AddCommand(tenant.Cmd)
 	Cmd.AddCommand(accesskey.Cmd)
+	Cmd.AddCommand(principal.Cmd)
+	Cmd.AddCommand(policy.Cmd)
 }