Describe the bug
When mirroring OCP release images to a self-hosted Quay registry with --remove-signatures=false, cosign signature manifests fail with HTTP 400 manifest invalid. The failure occurs consistently when a release image has been signed multiple times (producing identical duplicate layers in the .sig manifest).
Root Cause
putSignaturesToSigstoreAttachments in image/docker/docker_image_dest.go constructs a new OCI image config from scratch using imgspecv1.Image. It appends each signature layer's digest to ociConfig.RootFS.DiffIDs, then marshals that struct to produce the config blob.
The problem: RootFS.DiffIDs is defined by the OCI spec as uncompressed tar archive digests. The function instead writes cosign simplesigning payload digests (JSON blobs, not tar archives) into that field. Quay validates diff_ids against the actual layer content and rejects the manifest with 400.
Additionally, when the source .sig manifest contains duplicate identical layers (common when an image is signed across multiple release cycles), the function de-duplicates them — producing a 1-layer manifest from an 8-layer source. The constructed manifest no longer matches the source, so even a verbatim config blob from the source cannot save it.
The correct OCI/cosign convention is to use an empty config ({}), which is what the original source manifest carries.
Steps to Reproduce
- Mirror OCP release channels (e.g.
stable-4.18, stable-4.19, stable-4.20) to a self-hosted Quay registry using oc-mirror v2 with --remove-signatures=false
- Observe HTTP 400
manifest invalid errors during the signature push phase
- Fetch the source
.sig manifest from quay.io — it will have 8 identical layers and an empty config ({})
- Compare to what the library pushes — 1 layer, OCI image config with cosign payload digest in
diff_ids
Expected Behavior
The source .sig manifest (with its original config blob) is pushed verbatim to the destination registry. No manifest reconstruction occurs.
Actual Behavior
manifest invalid: manifest invalid
HTTP 400 from Quay on every .sig tag push. Spoke clusters report missing signatures.
Environment
|
|
| Library version |
go.podman.io/image/v5 v5.40.0 |
| Registry |
Red Hat Quay 3.16 |
| Consumer |
openshift/oc-mirror v4.22.7 |
| OCI content |
OCP release image cosign signatures (sha256-<digest>.sig tags) |
Proposed Fix
Replace the var ociConfig imgspecv1.Image block with var configBlob []byte, default it to []byte("{}") for new signatures, and fetch the existing config blob verbatim (without unmarshaling into a struct) for existing manifests. Remove the ociConfig.RootFS.DiffIDs = append(...) line and the json.Marshal(ociConfig) block entirely.
A working implementation is in PR #1017.
References
Describe the bug
When mirroring OCP release images to a self-hosted Quay registry with
--remove-signatures=false, cosign signature manifests fail with HTTP 400manifest invalid. The failure occurs consistently when a release image has been signed multiple times (producing identical duplicate layers in the.sigmanifest).Root Cause
putSignaturesToSigstoreAttachmentsinimage/docker/docker_image_dest.goconstructs a new OCI image config from scratch usingimgspecv1.Image. It appends each signature layer's digest toociConfig.RootFS.DiffIDs, then marshals that struct to produce the config blob.The problem:
RootFS.DiffIDsis defined by the OCI spec as uncompressed tar archive digests. The function instead writes cosign simplesigning payload digests (JSON blobs, not tar archives) into that field. Quay validatesdiff_idsagainst the actual layer content and rejects the manifest with 400.Additionally, when the source
.sigmanifest contains duplicate identical layers (common when an image is signed across multiple release cycles), the function de-duplicates them — producing a 1-layer manifest from an 8-layer source. The constructed manifest no longer matches the source, so even a verbatim config blob from the source cannot save it.The correct OCI/cosign convention is to use an empty config (
{}), which is what the original source manifest carries.Steps to Reproduce
stable-4.18,stable-4.19,stable-4.20) to a self-hosted Quay registry usingoc-mirror v2with--remove-signatures=falsemanifest invaliderrors during the signature push phase.sigmanifest fromquay.io— it will have 8 identical layers and an empty config ({})diff_idsExpected Behavior
The source
.sigmanifest (with its original config blob) is pushed verbatim to the destination registry. No manifest reconstruction occurs.Actual Behavior
manifest invalid: manifest invalid
HTTP 400 from Quay on every
.sigtag push. Spoke clusters report missing signatures.Environment
go.podman.io/image/v5 v5.40.0openshift/oc-mirrorv4.22.7sha256-<digest>.sigtags)Proposed Fix
Replace the
var ociConfig imgspecv1.Imageblock withvar configBlob []byte, default it to[]byte("{}")for new signatures, and fetch the existing config blob verbatim (without unmarshaling into a struct) for existing manifests. Remove theociConfig.RootFS.DiffIDs = append(...)line and thejson.Marshal(ociConfig)block entirely.A working implementation is in PR #1017.
References
containers/image:containers/image@7f71ddf(PR #2974, closed due to repo migration — this is the resubmission)