Importing images from OCI archive

[afbjorklund]

Currently you can load/import from an image list.

images.txt

It would be nice, if you could import an archive too.

images.tar

---

https://specs.opencontainers.org/image-spec/?v=v1.0.0

ctr images export

nerdctl save

Even docker archives should be compatible, these days.

[aceeric]

@afbjorklund I found this to be a useful exercise to experiment with using Claude.ai to implement a non-trivial feature into an existing project. At this time I have a new branch implementing this capability. Key points:

1. The load sub-command --image-file arg is now overloaded and accepts a text file or a tarball with auto-detection
2. The serve sub-command --preload-images is identically overloaded
3. There's an edge case. When an image tarball is created from a digest (not a tag) using the google crane utility for example it writes the RepoTags in the tarball like this: index.docker.io/grafana/grafana:i-was-a-digest. So for the load sub-command an additional arg is accepted --resolve-ref. To use this you load a single-image tarball (single means one repository in docker terminology, even if multi-arch), and you specify the full correct image ref using --resolve-ref. So if you had 100 tarballs with invalid tags you'd run the load command 100 times with two args. This didn't make sense in my opinion for the serve sub-command so only load supports it.

This feature doesn't meet any particular use case for me. I run the registry on my desktop as a systemd service 24x7x365 and whenever I stand up an ephemeral dev kubernetes cluster, I configure the cluster to pull through my registry.

However I did kick the tires a bit and it seemed to work. If you have the ability to build and run the server then maybe you can put it through some more rigorous testing before I merge it to main and tag it.

[afbjorklund]

It is mostly a "legacy" feature, being able to import from existing cache... They were either created by crane, or manually:

• Airgap tarballs for k8s images lima-vm/lima#4875

If implementing the feature fully, it could also make sense with a "save" feature that would do the opposite of "load".

i.e. take a list of images, and output an archive (for export).

[afbjorklund]

However I did kick the tires a bit and it seemed to work. If you have the ability to build and run the server then maybe you can put it through some more rigorous testing before I merge it to main and tag it.

It seems to work for some archives, and fail for others. Mostly the issue seems to be with the new "hybrid" archives.

"The archive implements both Docker Image Spec v1.2 and OCI Image Spec v1.0."

INFO[0000] skipping image (digest sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4): no usable ref

But with other tools, such as podman or crane, then it seems happy. But failing with the standard docker and containerd.

$ docker pull registry.k8s.io/pause:3.10.2 
3.10.2: Pulling from pause
81ede36234b0: Pull complete 
Digest: sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4
Status: Downloaded newer image for registry.k8s.io/pause:3.10.2
registry.k8s.io/pause:3.10.2
$ docker save registry.k8s.io/pause:3.10.2 >pause-docker.tar
$ docker load <pause-docker.tar
Loaded image: registry.k8s.io/pause:3.10.2

INFO[0000] loading images from tarball: pause-docker.tar 
INFO[0000] skipping image (digest sha256:2a7c0521faf29359bacce39c9a24522be4854d56e6096c959c9a9044a0dd4b14): no usable ref 
INFO[0000] loaded 0 images from tarball "pause-docker.tar" to the file system cache in 1.519394ms 

Experimented with multiple tools, seems that they all have their little quirks. But I only care about the OCI archives.

320K	pause-containerd.tar
320K	pause-crane.tar
732K	pause-docker.tar
324K	pause-nerdctl.tar
728K	pause-podman.tar

# some of the tools only support the legacy gzip, then it is not possible to compress...
# i.e. the layers are already compressed, so re-compressing the .tar doesn't help them
304K	pause-docker.tar.gz
272K	pause-docker.tar.zst

Support for RepoTags is therefore not as important as the image.ref.name. Would be nice to support the "hybrid" tar.
Unfortunately the OCI annotations look broken in docker/containerd , but I don't want to use the manifest.json...

docker

      "annotations": {
        "io.containerd.image.name": "registry.k8s.io/pause:3.10.2",
        "org.opencontainers.image.ref.name": "3.10.2"
      }

specification

      "annotations": {
        "org.opencontainers.image.ref.name": "registry.k8s.io/pause:3.10.2"
      }

---

--format=docker-archive

68b14877de21ededb2711dcda9ab929d66a57f9fe2d6abb9a1593372648ccea8.tar
4a83b15d3ecfe0d916b2d0a7991bc2854a629b8097017c2ee1ff65b30ae4c07c.json
f98f494423d69fc704d4d331bd5e3a52ebd5e1f08cbef449fbfc79dbc2ef6c0d/layer.tar
f98f494423d69fc704d4d331bd5e3a52ebd5e1f08cbef449fbfc79dbc2ef6c0d/VERSION
f98f494423d69fc704d4d331bd5e3a52ebd5e1f08cbef449fbfc79dbc2ef6c0d/json
manifest.json
repositories

--format=oci-archive

blobs/
blobs/sha256/
blobs/sha256/7ee3ff4149056be395e614c7d0c549da07d8982eb242a05f6a273180eb66c7d5
blobs/sha256/96765ebb24a18452ab86a478010c12c8459cd695f3e1a620542b0031d85333e6
blobs/sha256/cb84a9dbb358fcf03a07ea80c2a0d287d28129bf9ce8c46b922e8c40d945b395
index.json
oci-layout

[afbjorklund]

It would also need support from loading from stdin, so that you can pipe from zstdcat. Like with a - file name or so.

The tool itself only needs to support basic tar, maybe also tgz with the built-in go decompression (but that is optional)

[aceeric]

I fed your failure scenario (docker pull, docker save) to Claude and it fixed it. Updated branch pushed. (I had to rebase on main to pick up recent changes there for go vulnerabilities and then force-push this branch.) Honestly I'm going to have limited time to work on this over the next month due to other obligations. I don't know if you're a dev or not but you're welcome to use the LLM of your choice to see if you can get this functionally complete and submit a PR. Here's what I did to implement this new feature:

1. Create a new Claude project (I pay the monthly fee)
2. From this project upload all .go files from the cmd and impl directories preserving path/package structure
3. From the library project that this project uses, upload everything in the cmd, internal, and pkg directories preserving path/package structure
4. Write your requirements and indicate which requirements are currently implemented and which are not
5. Test & iterate

[afbjorklund]

Thanks for accepting the feature, it does not have to be fully implemented* at once of course... Happy to do a PR as well.

* The initial branch already did support both podman save --format docker-archive and podman save --format oci-archive.