Skip to content

Detect layer compression by content, not media type #405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattpollock wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Detect layer compression by content, not media type #405

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 34 additions & 5 deletions commands/rootfs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ package commands

import (
"archive/tar"
"bufio"
"bytes"
"compress/gzip"
"io"
"os"
Expand Down Expand Up @@ -195,26 +197,53 @@ func extractLayer(dest string, layer v1.Layer, bar *mpb.Bar, chown bool) error {

type compressReaderClose func() error

// Layer compression magic numbers.

@taylorsilva taylorsilva Jun 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: for easy lookup reference in the future link to https://en.wikipedia.org/wiki/List_of_file_signatures

Suggested change
// Layer compression magic numbers.
// Layer compression magic numbers. Can find more at:
// https://en.wikipedia.org/wiki/List_of_file_signatures

var (
zstdMagic = []byte{0x28, 0xb5, 0x2f, 0xfd}
gzipMagic = []byte{0x1f, 0x8b}
)

// compressionReader returns a tar.Reader over the (possibly compressed) layer.
//
// It detects the compression from the layer's own leading bytes rather than
// trusting the declared media type. Builders and registries don't always label
// zstd layers with the OCI media type (types.OCILayerZStd); BuildKit pushing
// with Docker schema2 emits application/vnd.docker.image.rootfs.diff.tar.zstd,
// which a media-type switch mis-routes to gzip and fails with
// "gzip: invalid header". Sniffing the magic bytes handles gzip, zstd, and
// uncompressed tar regardless of how the layer is labeled. mediaType is kept
// for signature compatibility but no longer drives the decision.
func compressionReader(layer io.ReadCloser, mediaType types.MediaType, bar *mpb.Bar) (*tar.Reader, compressReaderClose, error) {
Comment on lines +214 to 216

@taylorsilva taylorsilva Jun 26, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This func is private and only used in one place. Let's just update the signature and update where it's called, which should be higher up in this file.

Suggested change
// uncompressed tar regardless of how the layer is labeled. mediaType is kept
// for signature compatibility but no longer drives the decision.
func compressionReader(layer io.ReadCloser, mediaType types.MediaType, bar *mpb.Bar) (*tar.Reader, compressReaderClose, error) {
// uncompressed tar regardless of how the layer is labeled. mediaType is kept
// for signature compatibility but no longer drives the decision.
func compressionReader(layer io.ReadCloser, bar *mpb.Bar) (*tar.Reader, compressReaderClose, error) {

And update here:

registry-image-resource/commands/rootfs.go

Lines 83 to 91 in 1627112

mediaType, err := layer.MediaType()
if err != nil {
return err
}
tr, crc, err := compressionReader(lyr, mediaType, bar)
if err != nil {
return err
}

var cr io.Reader
var crc func() error

switch mediaType {
case types.OCILayerZStd:
reader, err := zstd.NewReader(bar.ProxyReader(layer))
br := bufio.NewReader(bar.ProxyReader(layer))
magic, err := br.Peek(4)
if err != nil && err != io.EOF {
return nil, nil, err
}

switch {
case bytes.HasPrefix(magic, zstdMagic):
reader, err := zstd.NewReader(br)
if err != nil {
return nil, nil, err
}
cr = reader
crc = func() error { reader.Close(); return nil }

default:
reader, err := gzip.NewReader(bar.ProxyReader(layer))
case bytes.HasPrefix(magic, gzipMagic):
reader, err := gzip.NewReader(br)
if err != nil {
return nil, nil, err
}
cr = reader
crc = func() error { return reader.Close() }

default:
// Uncompressed tar (or unknown) — read the stream as-is.
cr = br
crc = func() error { return nil }
}

tr := tar.NewReader(cr)
Expand Down