storage: preserve original compressed layer blob alongside extracted content

[sohankunkerkar]

Problem

Today, when a layer is pulled from a registry, applyDiff decompresses the layer into the overlay diff/ directory and discards the original compressed blob. Once that happens, the exact bytes corresponding to the registry digest are no longer available locally.

This isn't an issue for running containers, but it becomes a problem for workflows that need the original compressed blob again. As AI/ML images continue to grow, often into the tens of gigabytes, re-pulling just to get the original compressed form back becomes increasingly expensive.

Proposal

I'd like to add an optional PreserveCompressedBlob flag to LayerOptions.

When enabled, the original compressed layer would be saved alongside the existing layer metadata in the overlay storage. This would preserve the exact registry blob while still allowing the normal unpack flow to continue unchanged.

One detail worth noting: the preservation should be gated on whether the incoming stream is actually compressed (via archive.DetectCompression, which is already computed at the insertion point), rather than on whether OriginalDigest is set — there's a layer-reuse path where the digest is present but the stream is uncompressed.

The first consumer would be CRI-O, which could opt into this behavior during image pulls.

Why this approach?

The proposal is intentionally opt-in, so existing behavior remains unchanged unless a caller explicitly requests blob preservation.

It also fits naturally into the current storage model. The blob has the same lifetime as the layer itself, so normal image garbage collection automatically removes it. No additional cleanup mechanism is needed.

Initially this would only be supported by the overlay driver. Other drivers could opt in later if there's interest.

If this direction makes sense, I'm happy to put together a PR with the implementation details and discuss the API there.

cc @mtrmac @Luap99 @giuseppe @haircommander

[Luap99]

From a practical node raising an issue to then right creating a PR right away before we have the chance to comment here seems a bit pointless, either we keep the conversation if we want this here or on the PR. IF we say we don't want this then the PR would be a waste...

One detail worth noting: the preservation should be gated on whether the incoming stream is actually compressed (via archive.DetectCompression, which is already computed at the insertion point), rather than on whether OriginalDigest is set — there's a layer-reuse path where the digest is present but the stream is uncompressed.

I don't understand this, even an uncompressed layer still needs work to rebuild the full tar layer form the extract content. So you would still safe work by storing it as is.

Now one thing that I am missing here totally is the use case...

This isn't an issue for running containers, but it becomes a problem for workflows that need the original compressed blob again. As AI/ML images continue to grow, often into the tens of gigabytes, re-pulling just to get the original compressed form back becomes increasingly expensive.

That doesn't say anything about why this file is needed to be there locally. I can just guess you want to push this image again elsewhere?
Then having done a short peek at the PR you expose that path on the API level?! It feels wrong to expose such details to users at all, especially when there is no guarantee such files can even exist. What exactly will the caller do with this file?

[sohankunkerkar]

From a practical node raising an issue to then right creating a PR right away before we have the chance to comment here seems a bit pointless

Agreed, I should have waited for alignment here. Happy to close the PR and continue the discussion until we agree on the approach.

I don't understand this, even an uncompressed layer still needs work to rebuild the full tar layer from the extract content. So you would still save work by storing it as is.

Agreed that preserving reused uncompressed content would also save rebuild work — that could be a separate optimization. This guard is specifically about not mislabeling content as the compressed blob when it isn't. When a layer already exists in storage and is reused by diffID, createNewLayer reads it via store.Diff with noCompression, so the stream is always uncompressed. But in some cases it still sets OriginalDigest to the compressed blob digest from the manifest. Without the compression guard, we'd store uncompressed bytes and label them as the compressed blob matching that digest. Consumers expecting the registry-matching compressed form would get the wrong content.

Now one thing that I am missing here totally is the use case...
What exactly will the caller do with this file?

GetBlob in the storage source hardcodes noCompression as it always returns uncompressed bytes. That means skopeo copy --preserve-digests from local storage doesn't work: the copy pipeline has to recompress, and the digest changes. Having the original blob available fixes that for air-gapped mirroring workflows — pull once, push to internal mirrors without re-pulling from the source registry. Longer-term, this would also be a prerequisite for node-to-node P2P distribution tools (e.g. Spegel), which need to serve the original compressed bytes to peers.

[mtrmac]

Longer-term, this would also be a prerequisite for node-to-node P2P distribution tools (e.g. Spegel)

I continue to strongly think Spegel is strictly worse than an explicitly managed HA cluster of registries. Given that we have clearly insufficient capacity to handle every incoming item, I think known-undesirable ideas must be deprioritized over unknown-value ideas or prioritized ones.

[sohankunkerkar]

Just to clarify, I don't think this proposal is asking container-libs to support Spegel or any other P2P distribution protocol. The intent is much narrower. The only thing I'm proposing is a storage-level primitive to preserve the original compressed blob alongside the extracted layer, so that consumers can retrieve it later if they need it.

What a consumer does with that blob is outside the scope of container-libs. For example, if CRI-O wants to use it for P2P distribution, mirroring, or something else entirely, that integration would live in CRI-O, not here.

Regarding the HA registry point, I completely agree that HA registries address availability. The motivation I'm trying to solve is slightly different. In scenarios where many nodes pull very large images at the same time, bandwidth can still become the limiting factor even with a highly available registry. That said, I see that as a CRI-O use case rather than something container-libs itself needs to solve.

[mtrmac]

That turns the conversation from “CRI-O wants it for Spegel” into “CRI-O wants it for no specific reason”… not an improvement?

[sohankunkerkar]

That's fair. I wasn't trying to make Spegel the justification for the API. I mentioned it because it's what led me to look into this. The underlying capability is preserving the original compressed blob for callers that opt in. I think that stands on its own regardless of which consumer ends up using it.

[mtrmac]

the original compressed blob

(Also mentioned in #1012:) A single uncompressed layer can match several compressed blobs.

[phillebaba]

I thought I would jump in here as the creator of Spegel to share some of my thoughts. It is worth mentioning that Containerd allows you to configure this behavior with discard_unpacked_layers that is false by default. Spegel was built on top of the fact that the compressed layers were kept by Containerd. Which has over the years proven to be a light weight yet efficient way to both speed up container startup times and ensure image availability.

Funnily enough I first started investigating how (Podman, Buildah, and CRI-O) where storing image layers, and how it compared to Containerd, due to seeing differing image digests for images produced by different tools. Turns out Tar is not an idempotent process even when zeroing timestamps. Different implementations of tar can produce binary data that is slightly different but contains the same file. This is why if you pull an image with Podman and then push it to a new registry, the new copied image will have a different digests. This is generally a behavior that you don't want, especially as more and more people are pinning image references by digest.

While my opinion is probably biased, as I would like to see CRI-O support in Spegel, I do think that adding this feature would be an overall win for the container-libs ecosystem of tools, as they will be able to ensure that digests stay the same. I hope sharing some of my experiences gives some more reason to implement this 😄

[mtrmac]

Different implementations of tar can produce binary data that is slightly different but contains the same file. This is why if you pull an image with Podman and then push it to a new registry, the new copied image will have a different digests.

No, that is accounted for using tar-split.

The primary cause of differing digests is that compression is implementation-dependent and not even guaranteed to be deterministic within a single implementation. (And another episode of people who had “software running fine” encountering this is coming: https://tip.golang.org/doc/go1.27#compressflatepkgcompressflate )

[phillebaba]

Yeah sorry I got my terms mixed up. It was a while since I last looked at this. The fact that the output of compression is not consistently reproducable is frustrating. Especially considering that people build their supply chain security for OCI images based on expecting the whole merkle does not change so the index has the same digest. Which is why I think it would be valuable for the wider ecosystem.