feat(server): DB health + pool/webhook metrics endpoints (PR3/3)

[gavinelder]

Summary

PR 3 of 3 — final leg of the postgres enterprise hardening plan. Stacked on #65 → #64; please review/merge those first.

Adds four operational endpoints on the existing HTTP server, wires Kubernetes probes to them, and ships a postgres operations runbook.

Endpoint	Purpose	Probe target
GET /healthz	Process liveness, no DB dependency. Returns 200 whenever the server can respond.	livenessProbe
GET /healthz/db	Pings the pool with a 750ms timeout. 200 / 503.	readinessProbe
GET /metrics/db	pgxpool.Stat() snapshot as JSON.	(scrape)
GET /metrics/webhook	BatchServiceAdapter.GetMetrics() snapshot.	(scrape)

Routing is registered before ignoreUserAgentMiddleware so a probe client cannot be silenced by an operator's --ignored-user-agent flag. 503 responses reuse the existing ErrorResponse struct for consistency with the webhook handler.

manifests/deployment.yml gains livenessProbe (/healthz) and readinessProbe (/healthz/db) on the staticreg container.

Documentation

• docs/CONFIGURATION.md — adds Health/Metrics endpoints reference and a Production Checklist.
• docs/ARCHITECTURE.md — refreshes the database section with schema-isolation, goose migrations, and the new health/observability endpoints.
• docs/POSTGRES_OPERATIONS.md — new runbook covering: schema layout, inspecting state via SQL/HTTP, adding new migrations, rolling back via the goose CLI image, pool-tuning guidance, and the v0.7 → v0.8 data-consolidation procedure operators may want to run after PR1 lands.

Behavior change

Adding probes is opt-in for non-Kubernetes deployments (the YAML changes are inert if you don't use that manifest). For deployments using manifests/deployment.yml directly, k8s will start failing pods if /healthz/db returns 503 — this is the desired behavior (drains traffic from a pod whose DB connection is broken).

Test plan

• go build ./... && go vet ./... clean

• go test ./... (unit) passes — new tests for all four handler nil-paths plus the no-pool webhook-adapter happy path

• go test -tags=integration ./pkg/server/... passes against docker compose up postgres (verifies /healthz/db returns 200 and /metrics/db exposes the expected stat keys with a live pool)

• End-to-end smoke test: built binary, started against compose postgres, hit all four endpoints with curl. Confirmed 200 status and correct JSON shapes:

  /healthz         → {"status":"ok"}
  /healthz/db      → {"status":"ok"}
  /metrics/db      → {"acquired_conns":0,"idle_conns":3,"max_conns":25,...}
  /metrics/webhook → {"events_received":0,"events_dropped":0,...}
  

• Reviewer to verify a staging k8s rollout: probe failure when postgres is killed, recovery after restoration; /metrics/db consumed by an existing scraper.

🤖 Generated with Claude Code