Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,11 @@ jobs:

# ----------------------------------------------------------------------
# No-system-deps guarantee + packaging check. Fails if any forbidden
# numerical/BLAS crate has crept into the dependency tree, then dry-runs
# the publishable crates to catch packaging problems before a real release.
# numerical/BLAS crate has crept into the dependency tree, then checks the
# publishable crate artifacts before a real release. The lockstep
# ordvec-manifest crate cannot run a registry-backed dry-run for a bumped
# version until ordvec itself is published, so CI verifies the pre-publish
# package artifact that release.yml later byte-compares after the core publish.
# ----------------------------------------------------------------------
deps:
name: deps (no-system-deps + publish dry-run)
Expand Down Expand Up @@ -258,21 +261,22 @@ jobs:
# from the current directory. The Python binding remains publish = false
# and ships to PyPI via maturin.
run: cargo publish -p ordvec --dry-run --locked
- name: cargo publish --dry-run -p ordvec-manifest
run: cargo publish -p ordvec-manifest --dry-run --locked
- name: cargo package --no-verify -p ordvec-manifest
run: cargo package -p ordvec-manifest --locked --no-verify

# ----------------------------------------------------------------------
# Pin the release-publish invariants. release.yml is tag-triggered (with the
# two registry publishes gated behind GitHub Environments), so its release-
# registry publishes gated behind GitHub Environments), so its release-
# specific flow runs only on a real release. Two structural lints guard it
# on every push/PR so regressions can't sneak in between releases:
# * release_publish_invariants.sh — the *.cdx.json SBOM never reaches PyPI
# (a generated SBOM once broke both publish paths).
# * release_signed_release_invariants.sh — the signed-release / provenance
# graph stays intact: release-assets-draft stays draft, the SLSA
# generator emits a .intoto.jsonl, both publishes need the draft assets,
# publish-crate proves byte-identity vs the attested .crate, and
# publish-github-release un-drafts ONLY after both publishes succeed.
# generator emits a .intoto.jsonl, registry publishes need the draft
# assets, both Rust crates prove byte-identity vs their attested .crate
# files, and publish-github-release un-drafts ONLY after all registry
# publishes succeed.
# ----------------------------------------------------------------------
release-guard:
name: release-publish invariants
Expand Down
218 changes: 181 additions & 37 deletions .github/workflows/release.yml

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,8 +174,8 @@ candidate slices passed to `Search` until the call returns.
[`docs/determinism.md`](https://github.com/Fieldnote-Echo/ordvec/blob/main/docs/determinism.md),
[`THREAT_MODEL.md`](https://github.com/Fieldnote-Echo/ordvec/blob/main/THREAT_MODEL.md)
- **Manifest verifier, C ABI, and Go wrapper:**
`ordvec-manifest` is versioned in lockstep with the core crate and is
package-gated separately; use the GitHub checkout for `ordvec-ffi/`,
`ordvec-manifest` is versioned and published in lockstep with the core crate
through its own package gate; use the GitHub checkout for `ordvec-ffi/`,
`ordvec-go/`, and
[`docs/c-api.md`](https://github.com/Fieldnote-Echo/ordvec/blob/main/docs/c-api.md).
- **Pre-1.0 compatibility policy:**
Expand Down
110 changes: 64 additions & 46 deletions RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
# Releasing `ordvec`

> **Publish is held.** A real `cargo publish` / PyPI publish happens only on
> **Publish is held.** A real crates.io / PyPI publish happens only on
> the maintainer's explicit approval. CI never publishes — the unified release
> pipeline builds, attests, and attaches everything to the GitHub Release
> automatically on a tag push, then **waits at the `crates-io` and `pypi`
> environment gates** for a required-reviewer approval before either registry
> environment gates** for a required-reviewer approval before any registry
> push.

`ordvec` (the Rust crate) and `ordvec` on PyPI (the PyO3 wheel built from
`ordvec-python/`) are released by **pushing a `vMAJOR.MINOR.PATCH` tag** to a
commit on `main`. The release workflow handles build, canonical Python artifact
selection, attestation, SLSA provenance, Release-asset attach, and un-draft
automatically; only the two registry gates are manual.
`ordvec` (the Rust crate), `ordvec-manifest` (the lockstep manifest verifier
crate), and `ordvec` on PyPI (the PyO3 wheel built from `ordvec-python/`) are
released by **pushing a `vMAJOR.MINOR.PATCH` tag** to a commit on `main`. The
release workflow handles build, canonical Python artifact selection,
attestation, SLSA provenance, Release-asset attach, and un-draft automatically;
only the registry environment approvals are manual.

## Release pipeline controls

Expand All @@ -24,7 +25,7 @@ The unified `release.yml`:
`main` for the tagged SHA — `ci.yml`, `python.yml`, `fuzz.yml`, `codeql.yml`
(a *successful* run for that exact SHA on `main`);
- publishes via **OIDC trusted publishing** (no long-lived crates.io / PyPI
tokens in the repo);
tokens in the repo) for both Rust crates and the Python distribution;
- canonicalizes the Python dist before attestation and release upload: for a
new PyPI version it uses the current run's wheels/sdist; if PyPI already owns
that immutable version during recovery, it downloads the exact PyPI-served
Expand All @@ -35,31 +36,41 @@ The unified `release.yml`:
**before** the gated publishes — a failed attestation fails the release
closed, so nothing ships without provenance recorded. In recovery mode where
PyPI files already exist, the GitHub/SLSA subjects are deliberately limited
to the crate built by the current run; the Python files are verified immutable
PyPI bytes from the earlier Trusted Publishing upload, not falsely claimed as
rebuilt by the recovery run;
- stages the **`.crate`, canonical wheels, canonical sdist, `*.sigstore.json` bundle, and
`*.intoto.jsonl` provenance** on the GitHub Release while it is still **a
to the Rust crates built by the current run; the Python files are verified
immutable PyPI bytes from the earlier Trusted Publishing upload, not falsely
claimed as rebuilt by the recovery run;
- stages the **`.crate` files, canonical wheels, canonical sdist,
`*.sigstore.json` bundle, and `*.intoto.jsonl` provenance** on the GitHub Release while it is still **a
DRAFT** (`release-assets-draft` is the sole Release-asset writer — no manual
attach, which is what v0.2.0's manual step missed);
- proves **byte-identity** in `publish-crate` on both sides of `cargo publish`:
- proves **byte-identity** in `publish-crate` and `publish-manifest-crate` on
both sides of `cargo publish`:
1. **pre-publish gate** — downloads the SLSA-attested `.crate` artifact,
re-packages with `--locked`, and `sha256`-compares before minting the
crates.io OIDC token. Defends against toolchain drift / deterministic-
packaging regression; if they differ, fails closed **before** the
token is minted (nothing reaches crates.io);
2. **post-publish empirical proof** — downloads the just-published `.crate`
from `crates.io/api/v1/crates/ordvec/<v>/download` and `sha256`-compares
to the attested artifact. `cargo publish` runs its own internal
packaging step the pre-publish gate cannot inspect; this is the only
check that proves the bytes crates.io actually serves equal the SLSA-
attested bytes. A mismatch fails closed, so `publish-github-release`
never un-drafts the Release (the version is then yank-only on
crates.io, but the failure is loudly observable);
- **un-drafts the GitHub Release ONLY after BOTH `publish-crate` AND
`publish-pypi` succeed** (`publish-github-release` is the sole un-draft
point). If either publish fails or is skipped, the Release stays DRAFT — no
public Release ever exists for a version the registries refused;
from `crates.io/api/v1/crates/<crate>/<v>/download` and
`sha256`-compares to the attested artifact. `cargo publish` runs its own
internal packaging step the pre-publish gate cannot inspect; this is the
only check that proves the bytes crates.io actually serves equal the
SLSA-attested bytes. A mismatch fails closed, so
`publish-github-release` never un-drafts the Release (the version is
then yank-only on crates.io, but the failure is loudly observable);
- publishes `ordvec-manifest` only after the lockstep `ordvec` crate has
published and passed its crates.io-served byte-identity readback. The
pre-publish manifest `.crate` artifact is created with `cargo package
--no-verify` because Cargo cannot verify a lockstep dependency version before
`ordvec` exists on crates.io; `publish-manifest-crate` then runs the normal
verified `cargo package -p ordvec-manifest --locked` after `ordvec` is
published and byte-compares that output to the attested artifact before
minting its own OIDC token;
- **un-drafts the GitHub Release ONLY after `publish-crate`,
`publish-manifest-crate`, AND `publish-pypi` succeed**
(`publish-github-release` is the sole un-draft point). If any publish fails
or is skipped, the Release stays DRAFT — no public Release ever exists for a
version the registries refused;
- pins every third-party action by **commit SHA** (the one mandated exception
is the SLSA reusable workflow, tag-pinned per SLSA's trust model), sets
`persist-credentials: false`, and defaults to `permissions: contents: read`.
Expand Down Expand Up @@ -93,11 +104,15 @@ the GitHub Release.
### Trusted-publisher configuration (one-time, in the registries)

The crates.io and PyPI Trusted Publisher records must point at this workflow
filename. Until either is updated, the corresponding gated publish fails
filename. Until a record is updated, the corresponding gated publish fails
**closed** at the OIDC exchange (no risk of a bad publish; just a failed run).

- **crates.io** → `ordvec` → Settings → Trusted Publishing → GitHub publisher:
`workflow = release.yml`, `environment = crates-io`.
- **crates.io** → `ordvec-manifest` → Settings → Trusted Publishing → GitHub
publisher: `workflow = release.yml`, `environment = crates-io`. If crates.io
requires an initial owner bootstrap before a new crate's Trusted Publisher can
be configured, do that explicit maintainer-approved bootstrap before tagging.
- **PyPI** → `ordvec` → Publishing → GitHub publisher: `workflow = release.yml`,
`environment = pypi`.

Expand Down Expand Up @@ -167,8 +182,9 @@ filename. Until either is updated, the corresponding gated publish fails

This verifies the GitHub Environments still require the expected reviewer
and accept only the stable release tag pattern. Separately verify the
registry Trusted Publisher records by hand: crates.io must point to
`release.yml` / `crates-io`, and PyPI must point to `release.yml` / `pypi`.
registry Trusted Publisher records by hand: crates.io must point both
`ordvec` and `ordvec-manifest` to `release.yml` / `crates-io`, and PyPI must
point `ordvec` to `release.yml` / `pypi`.
6. Get the maintainer's explicit go to publish.
7. Push the version tag from `main` (signed):

Expand All @@ -177,30 +193,32 @@ filename. Until either is updated, the corresponding gated publish fails
git push origin vX.Y.Z
```

`release.yml` triggers automatically. It builds the `.crate`, wheels, and
sdist; selects the canonical Python dist (current build for a new PyPI
version, verified PyPI bytes for an existing immutable version); attests the
files this run can honestly attest (GitHub attestation store +
`release.yml` triggers automatically. It builds the two `.crate` files,
wheels, and sdist; selects the canonical Python dist (current build for a
new PyPI version, verified PyPI bytes for an existing immutable version);
attests the files this run can honestly attest (GitHub attestation store +
`*.sigstore.json`); generates the SLSA `*.intoto.jsonl`; and stages every
artifact, the attestation bundle, and the provenance on the GitHub Release
— **as a DRAFT**. It then pauses at the two registry environment gates.
8. **Approve the two publish environments** when they pause in the Actions UI
(one for `crates-io`, one for `pypi`). The required-reviewer approval is
what authorises the registry push.
- `publish-crate` first sha256-compares its repackaged `.crate` to the
SLSA-attested artifact — if they diverge (toolchain drift, etc.) the job
fails closed BEFORE the OIDC token is minted, so nothing reaches
crates.io. Re-run / investigate.
- Once **both** registry gates succeed, `publish-github-release` un-drafts
the GitHub Release automatically. If one gate fails, the Release stays
DRAFT — investigate and re-run from a fixed workflow rather than approving
the other registry into another partial state.
— **as a DRAFT**. It then pauses at the registry environment gates.
8. **Approve each publish environment pause** in the Actions UI. There are
three registry publish jobs: `publish-crate`, `publish-manifest-crate`, and
`publish-pypi`. The two crates.io jobs use the same `crates-io` environment
and may require separate approvals; PyPI uses the `pypi` environment.
Required-reviewer approval is what authorises each registry push.
- `publish-crate` and `publish-manifest-crate` first sha256-compare their
repackaged `.crate` to the SLSA-attested artifact — if either diverges
(toolchain drift, etc.) the job fails closed BEFORE the OIDC token is
minted, so nothing reaches crates.io. Re-run / investigate.
- Once **all** registry publish jobs succeed, `publish-github-release`
un-drafts the GitHub Release automatically. If one gate fails, the Release
stays DRAFT — investigate and re-run from a fixed workflow rather than
approving another registry into a partial state.
- `publish-pypi` either uploads the fresh canonical dist or, if PyPI already
serves that version, skips upload and verifies the existing files. In both
modes it compares every PyPI-served wheel/sdist SHA-256 digest against the
canonical `dist/` files before the GitHub Release can un-draft.
9. Verify each published artifact and its provenance:
- crates.io / docs.rs;
- crates.io / docs.rs for `ordvec` and `ordvec-manifest`;
- PyPI (confirm the post-publish hash-verification log, optionally
`pip download ordvec==X.Y.Z` and inspect, plus check the PEP 740 attestation
at `GET https://pypi.org/integrity/ordvec/X.Y.Z/<file>/provenance`);
Expand All @@ -212,6 +230,6 @@ filename. Until either is updated, the corresponding gated publish fails

## Coordinated release note

The crate publish, the PyPI wheel, and the paper's Zenodo deposit are
The Rust crate publishes, the PyPI wheel, and the paper's Zenodo deposit are
coordinated (the paper consumes the bindings for a final cold-repro run). Do
not ship one leg in isolation without the maintainer's go.
25 changes: 14 additions & 11 deletions THREAT_MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,14 +307,16 @@ trust model requires be pinned by version *tag*); `persist-credentials: false`
on every checkout; `permissions: contents: read` default. The **release
workflow** (`release.yml`) is tag-triggered with a strict-SemVer guard; build,
GitHub attestation, SLSA provenance, Release-asset attach, and un-draft all
run automatically, while the **`crates.io`** and **`pypi`** publishes are
gated behind GitHub Environments with **Required reviewers** (the only manual
step). It runs a `require-ci-green` gate against `main`, publishes via **OIDC
trusted publishing** (no long-lived registry tokens), and emits **SLSA build
run automatically, while the two **`crates.io`** publish jobs (`ordvec` first,
then lockstep `ordvec-manifest`) and the **`pypi`** publish job are gated
behind GitHub Environments with **Required reviewers** (the only manual step).
It runs a `require-ci-green` gate against `main`, publishes via **OIDC trusted
publishing** (no long-lived registry tokens), and emits **SLSA build
provenance** (`actions/attest-build-provenance` + a `slsa-github-generator`
`*.intoto.jsonl` attached to the GitHub Release) **before** publish — a failed
attestation fails the release closed. PyPI additionally gets **PEP 740**
attestations via Trusted Publishing.
attestation fails the release closed. Each Rust publish job proves pre- and
post-publish crates.io byte identity against the attested `.crate`; PyPI
additionally gets **PEP 740** attestations via Trusted Publishing.

**Static / supply-chain analysis:** **CodeQL** scans Rust, Python, and Actions
(no-build databases); **OpenSSF Scorecard** publishes SARIF to code scanning
Expand All @@ -336,7 +338,7 @@ branch-only allowlist would deadlock publishing — see RELEASING.md). The
push-event CI run on `main`, and `main` itself is branch-protected (PR review,
no force-push) — so a release cannot be cut from an unmerged or attacker
branch, and no publish runs without an explicit human approval. The remaining residual is *maintainer-account compromise*: a
single owner both cuts the release tag and approves both publishes, so account takeover (or social
single owner both cuts the release tag and approves all publishes, so account takeover (or social
engineering) is not caught by a second human. *Mitigations:* strong 2FA /
passkeys on the maintainer account; recruiting a **second owner/maintainer**
(also an open OpenSSF Best-Practices item) — which would additionally make a
Expand Down Expand Up @@ -517,10 +519,11 @@ x86_64/aarch64 unsafe paths (SIMD-004).
in the Rust and Python API docs (THREAT-QUERY-001).

**Later (not release blockers):** a second maintainer/owner (then a release
wait timer becomes meaningful); an optional sidecar index verifier
(`ordvec verify` / external HMAC/BLAKE3 manifest) if a deployment requires
tamper-evidence (DESER-002); a `safe_copy=True` FFI isolation option
(FFI-001); layering ASAN onto the Intel SDE AVX-512 leg.
wait timer becomes meaningful); stronger deployment-specific manifest
trust-policy UX such as external signatures/HMACs if a deployment requires
tamper-evidence beyond `ordvec-manifest`'s hash-bound sidecar verification
(DESER-002); a `safe_copy=True` FFI isolation option (FFI-001); layering ASAN
onto the Intel SDE AVX-512 leg.

---

Expand Down
Loading
Loading