Change the fake config in sigstore attachments - #1017
Conversation
These changes fix the problem with openshift/oc-mirror#1478 Signed-off-by: Michael <58675276+Squid-Bomb@users.noreply.github.com>
There was a problem hiding this comment.
Verified this fix against the root cause.
The original code declared var ociConfig imgspecv1.Image, set RootFS.Type = "layers", and appended each signature layer's digest to RootFS.DiffIDs before marshaling. The OCI spec defines diff_ids as uncompressed tar archive digests — but cosign signature layers are simplesigning JSON payloads, not tar archives. Quay validates this and rejects the manifest with HTTP 400 manifest invalid.
Additionally, when a .sig manifest has duplicate identical layers (common for images signed across multiple OCP release cycles), the existing code de-duplicated them before building the new manifest. The resulting 1-layer manifest no longer matched the source, making recovery impossible even with the correct config blob.
This fix is correct: configBlob = []byte("{}") for new manifests follows the cosign standard, and fetching the existing config blob verbatim (without unmarshaling into a struct) for existing manifests preserves the original without modification.
Tested against OCP stable-4.18/4.19/4.20 mirrored to self-hosted Quay 3.16 — zero manifest invalid errors after applying this change. An equivalent fix was previously merged to containers/image at commit 7f71ddf (PR #2974) but was lost in the repo migration.
Squid-Bomb
left a comment
There was a problem hiding this comment.
This has been tested and has been proven to work
Summary
Bump
github.com/containers/image/v5to pick up the sigstore attachmentconfig fix in containers/image@7f71ddf,
which resolves
manifest invaliderrors when mirroring cosign signatureOCI artifacts to self-hosted Quay registries.
Closes #1478
Problem
When mirroring OCP release images to a self-hosted Quay registry with
--remove-signatures=false, cosign signature manifests (sha256-*.sig)fail to push with HTTP 400
manifest invalidfor any component imagewhose sig manifest contains repeated identical layer digests.
This affects OCP component images that have been signed multiple times
(e.g. images shared across multiple OCP releases), which accumulate one
identical cosign layer per signing operation. A typical affected manifest
has 8 layers all with the same digest.
The error surfaces as:
[ERROR]: [Worker] error mirroring image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:
error: writing signatures: uploading manifest sha256-.sig to /platform/openshift/release: manifest invalid
Root Cause
In
putSignaturesToSigstoreAttachments(docker/docker_image_dest.go incontainers/image), when a sig manifest has duplicate layer digests, all
but the first are detected as duplicates and skipped. The function then
constructs a new 1-layer manifest with a freshly serialized OCI image
config containing:
{"rootfs": {"type": "layers", "diff_ids": ["sha256:<cosign-payload-digest>"]}} diff_ids is an OCI image concept for uncompressed tar layer digests. The value written here is the digest of a 334-byte cosign JSON payload, not a tar archive. Quay validates diff_ids entries against the OCI image spec and rejects the manifest. Manually pushing the verbatim source manifest (which uses a correct empty {} config, the cosign standard) succeeds with HTTP 201, confirming the issue is in how containers/image constructs the config — not in Quay or the manifest content itself. Fix The upstream fix in containers/image@7f71ddf removes the intermediate imgspecv1.Image struct entirely. For new sig manifests it uses []byte("{}") directly (the correct cosign empty config). For existing manifests it fetches the config blob verbatim without unmarshalling/remarshalling, preserving whatever the registry already has. The DiffIDs append and final json.Marshal are removed. Testing Mirrored OCP stable-4.18 (4.18.33–4.18.48), stable-4.19, and stable-4.20 channels to a self-hosted Quay 3.16 instance Confirmed zero manifest invalid errors across all sig manifest pushes Spoke clusters validated image signatures successfully post-sync Reproduced the failure on the previous vendor version and confirmed it is resolved with this bump