lib: Add experimental unified storage support for install #1601

jmarrero · 2025-09-09T17:28:09Z

lib: Add experimental unified storage support for install

Add an experimental --experimental-unified-storage flag to bootc install
that uses bootc's container storage (/usr/lib/bootc/storage) to pull
images first, then imports from there. This is the same approach used
for logically bound images (LBIs).

Background:
The unified storage approach allows bootc to share container images with
podman's storage, reducing disk space and enabling offline installs when
images are pre-pulled to the host's container storage.

Changes:

Add --experimental-unified-storage CLI flag to install subcommands
Add sysroot_path parameter to prepare_for_pull_unified() and pull_unified()
to handle the different mount points during install vs upgrade/switch
Handle localhost images specially by exporting from ostree to container
storage first (these can't be pulled from a registry)
Skip pull in prepare_for_pull_unified() if image already exists in
bootc storage
Add TMT test for install with unified storage flag
Add TMT test for switching to unified storage on running system
Add integration test for system-reinstall-bootc with unified storage

The sysroot_path fix is needed because during install the target disk
is mounted at a specific path (e.g., /var/mnt), not /sysroot. Skopeo
needs the actual filesystem path to find the bootc storage.

Relates: #20

Assisted-by: Claude Code (Sonnet 4.5 & Opus 4.5)

[core@cosa-devsh ~]$ sudo podman --storage-opt=additionalimagestore=/usr/lib/bootc/storage images
REPOSITORY                       TAG         IMAGE ID      CREATED       SIZE
quay.io/jmarrero_rh/soft-reboot  1           97f84fcb062e  25 hours ago  1.83 GB

cgwalters

Thanks for working on this!

cgwalters · 2025-09-09T18:20:03Z

crates/lib/src/cli.rs

+    /// Use podman/skopeo to pull image to additionalimagestore, then read from container storage.
+    /// This provides a unified approach that leverages existing container tooling.
+    #[clap(long)]
+    pub(crate) unified: bool,


I don't think this should be a CLI flag that operates just once; it should be something like bootc image --set-unified or so and act persistently.

Also we should support setting this at install time so that it happens from the very start.

(hmm my bad I thought we discussed this but I failed to update the issue #20 or something maybe?)

Ahh man yeah, we talked about saving the config on the origin file IIRC, I forgot.

cgwalters · 2025-09-09T18:20:41Z

crates/lib/src/podman.rs

+    use bootc_utils::CommandRunExt;
+
+    // Use podman pull with additionalimagestore pointing to bootc storage
+    let bootc_storage_path = "/usr/lib/bootc/storage";


Use podstorage.rs instead please

To elaborate on this, an important aspect here is currently the GC of the bootc/storage instance is rooted in the set of LBIs. That will need to be extended to include the host image.

Another big task related to this - we'll eventually really want to use the podman HTTP APIs so we can properly monitor progress. Right now we are very crudely relying on doing this synchronously to the existing tty and reusing the podman CLI tty output. But that can't really work if we ever need to render our own progress APIs.

https://lib.rs/crates/bollard might be a good choice for this?

cgwalters · 2025-09-09T18:21:33Z

crates/lib/src/deploy.rs

+    match prepare_for_pull_unified(repo, imgref, target_imgref, store).await? {
+        PreparedPullResult::AlreadyPresent(existing) => {
+            // Log that the image was already present (Debug level since it's not actionable)
+            const IMAGE_ALREADY_PRESENT_ID: &str = "5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f9";


Note that if we go this route, we should also not log to the journal in the ostree pull path because otherwise we're double logging.

cgwalters · 2025-09-09T18:24:07Z

crates/lib/src/cli.rs

+        let fetched = if opts.unified {
+            crate::deploy::pull_unified(repo, imgref, None, opts.quiet, prog.clone(), sysroot).await?
+        } else {
+            crate::deploy::pull(repo, imgref, None, opts.quiet, prog.clone()).await?


This relates to #1599 (comment)

Basically how about changing this code to take a Storage which would hold the persistent flag, and then crate::deploy::pull would itself query that flag and change its behavior.

That would also implicitly then fix the install path to behave the same.

jlebon · 2025-10-07T13:31:09Z

Do we get reflinks when importing from the alternate storage into the ostree repo?

cgwalters · 2025-10-07T14:36:03Z

Do we get reflinks when importing from the alternate storage into the ostree repo?

#20 links to containers/container-libs#144

jlebon · 2025-10-07T15:42:36Z

Do we get reflinks when importing from the alternate storage into the ostree repo?

#20 links to containers/container-libs#144

Hmm, that's about copying between two container storages, but I was asking about the ostree import. But #20 does say:

From there, we do a reflink/hardlink copy into the ostree store (i.e. This would duplicate metadata, but not data on disk. Reflinks should always work.

Which kinda answers the question.

Basically without reflinks into the ostree storage, IMO this makes it a lot less obvious that this new option is objectively better and it instead becomes a discussion of trade-offs.

jmarrero · 2025-12-02T20:06:50Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental support for unified storage, allowing bootc to pull images into its own container storage before importing them. This is a significant feature that touches installation, upgrade, and switch paths. The changes include new CLI options, a new image set-unified command, and the core logic for the unified pull mechanism.

The implementation is comprehensive and includes new integration tests. My review has identified a few areas for improvement:

A bug in the image reference formatting for unified storage checks during installation.
Inconsistent error handling when checking for images in the unified store.
A potential panic due to an .unwrap() call on a Result.

Addressing these points will improve the robustness and maintainability of this new feature.

crates/lib/src/image.rs

crates/lib/src/install.rs

crates/lib/src/deploy.rs

cgwalters · 2025-12-02T20:51:58Z

On the progress status side maybe we can use https://lib.rs/crates/bollard to talk to podman and I think it should help us get data out of the events?

crates/lib/src/image.rs

jmarrero · 2025-12-10T17:33:10Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental support for unified storage in bootc, a significant and well-implemented feature. The changes are comprehensive, adding the --experimental-unified-storage flag to install and switch commands, auto-detection for upgrade, and a new bootc image set-unified command for onboarding existing systems. The implementation correctly handles different sysroot paths and edge cases like localhost images. The inclusion of integration and TMT tests demonstrates a thorough approach to quality assurance. My review includes a few minor suggestions to improve code style and readability.

crates/lib/src/deploy.rs

jmarrero · 2025-12-10T17:44:27Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental unified storage support for bootc install and related commands. The changes are extensive and well-thought-out, adding a new --experimental-unified-storage flag and the bootc image set-unified command. The implementation correctly handles different scenarios, such as install versus upgrade, and the special case of images originating from containers-storage. The addition of comprehensive integration and TMT tests is commendable. I've identified one high-severity logical issue in the implementation of bootc image set-unified where it could operate on a staged image instead of the booted one. I've also noted a correct and important bug fix in the image pruning logic. Overall, this is a solid contribution that adds a valuable feature.

crates/lib/src/image.rs

cgwalters

Thanks! A first pass review here, this is looking quite plausible!

cgwalters · 2025-12-10T19:15:53Z

crates/lib/src/cli.rs

+    /// When enabled, this uses bootc's container storage (/usr/lib/bootc/storage) to pull
+    /// the image first, then imports it from there. This is the same approach used for
+    /// logically bound images.
+    #[clap(long = "experimental-unified-storage")]


I think this should be hide = true

cgwalters · 2025-12-10T19:18:03Z

crates/lib/src/deploy.rs

+pub(crate) async fn should_use_unified_storage(
+    store: &Storage,
+    imgref: &ImageReference,
+    explicit_flag: Option<bool>,


This seems strange to me; nothing AFAICS passes Some(false). And if Some(true) is passed the caller will just get back true. So why even call this?

cgwalters · 2025-12-10T19:22:52Z

crates/lib/src/image.rs

+async fn image_exists_in_host_storage(image: &str) -> Result<bool> {
+    use tokio::process::Command as AsyncCommand;
+    let mut cmd = AsyncCommand::new("podman");
+    cmd.args(["image", "exists", image]);


I've never been a fan of using exit codes to communicate things like this. Not a blocker but we should definitely try to cut over to using bollard or so.

cgwalters · 2025-12-10T19:23:17Z

crates/lib/src/image.rs

+    //         local container storage. The image only exists in ostree now, so we need
+    //         to export from ostree to bootc storage.
+    // Case 2: Otherwise, pull from the specified transport (usually a remote registry).
+    let is_containers_storage = imgref.transport == "containers-storage";


We have a Transport enum for this

cgwalters · 2025-12-10T19:24:38Z

crates/lib/src/utils.rs


+/// Converts an ImageReference to a container reference string suitable for use with container storage APIs.
+/// For registry transport, returns just the image name. For other transports, prepends the transport.
+pub(crate) fn imageref_to_container_ref(imgref: &crate::spec::ImageReference) -> String {


Seems cleaner to have this as a method impl ImageReference { fn to_short_container_ref() ?

cgwalters · 2025-12-10T20:07:41Z

crates/lib/src/image.rs

+        // Check if the image already exists in the default containers-storage.
+        // This can happen if someone did a local build (e.g., podman build) and
+        // we don't want to overwrite it with an export from ostree.
+        ensure_floating_c_storage_initialized();


I think we can put this global init somewhere farther up?

cgwalters · 2025-12-10T20:08:29Z

crates/lib/src/image.rs

+            // We need to export from ostree to default containers-storage (/var/lib/containers)
+            tracing::info!("Image not found in containers-storage; exporting from ostree");
+            let booted = host.status.booted.as_ref().unwrap();
+            let booted_image = booted.image.as_ref().unwrap();


Hmm I think we can avoid these unwraps by using the first arg from crate::status::get_status_require_booted(ostree)?; above

cgwalters · 2025-12-10T20:08:59Z

crates/lib/src/image.rs

+            };
+
+            let mut opts = ostree_ext::container::store::ExportToOCIOpts::default();
+            opts.progress_to_stdout = true;


Let's add a // TODO: bridge to progress API here

cgwalters · 2025-12-10T20:10:37Z

crates/system-reinstall-bootc/src/main.rs

    // Note if we ever add any other options here,
    #[arg(long)]
    pub(crate) composefs_backend: bool,
+    #[arg(long = "experimental-unified-storage")]


Ditto re hide = true

cgwalters · 2025-12-10T20:11:22Z

crates/lib/src/image.rs

+    let sysroot = crate::cli::get_storage().await?;
+    let ostree = sysroot.get_ostree()?;


I think this stuff should be passed down from the caller, that's how most of the other logic works

jmarrero · 2025-12-11T03:45:06Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces experimental support for unified storage in bootc install, which allows sharing container images with podman's storage to reduce disk usage and enable offline installations. The changes include a new --experimental-unified-storage flag, modifications to upgrade and switch to auto-detect and use unified storage, and special handling for localhost images. New integration and TMT tests are also added to cover these new code paths.

My review focuses on correctness and maintainability. I've identified a couple of areas for improvement:

An inconsistency in the image pulling strategy for unified storage, which could lead to unnecessary network operations.
A small opportunity to simplify code by removing a redundant check, improving readability and reflecting the guarantees provided by other functions.

Overall, the changes are well-structured and the addition of tests is great. Addressing these points will improve the robustness and maintainability of the new feature.

crates/lib/src/deploy.rs

gemini-code-assist · 2025-12-11T03:47:02Z

crates/lib/src/image.rs

+    let booted_entry = host
+        .status
+        .booted
+        .as_ref()
+        .ok_or_else(|| anyhow::anyhow!("No booted deployment found"))?;


The get_status_require_booted function guarantees that host.status.booted is Some. Therefore, using ok_or_else here is unnecessary and can be simplified. Using expect() would be more idiomatic and clearly state the assumption that this should never be None.

Suggested change

let booted_entry = host

.status

.booted

.as_ref()

.ok_or_else(|| anyhow::anyhow!("No booted deployment found"))?;

let booted_entry = host

.status

.booted

.as_ref()

.expect("No booted deployment found; this should be guaranteed by get_status_require_booted");

Add an experimental --experimental-unified-storage flag to bootc install that uses bootc's container storage (/usr/lib/bootc/storage) to pull images first, then imports from there. This is the same approach used for logically bound images (LBIs). Background: The unified storage approach allows bootc to share container images with podman's storage, reducing disk space and enabling better integration with podman. Changes: - Add --experimental-unified-storage CLI flag to install subcommands - Add sysroot_path parameter to prepare_for_pull_unified() and pull_unified() to handle the different mount points during install vs upgrade/switch - Handle container-storage transport - Skip pull in prepare_for_pull_unified() if image already exists in bootc storage - Add TMT test for install with unified storage flag - Add TMT test for switching to unified storage on running system The sysroot_path fix is needed because during install the target disk is mounted at a specific path (e.g., /var/mnt), not /sysroot. Skopeo needs the actual filesystem path to find the bootc storage. Relates: bootc-dev#20 Assisted-by: Claude Code (Sonnet 4.5) Signed-off-by: Joseph Marrero Corchado <[email protected]>

bootc-bot bot requested a review from cgwalters September 9, 2025 17:28

cgwalters reviewed Sep 9, 2025

View reviewed changes

jmarrero force-pushed the unified-storage-wip branch from 1baf1e8 to 9c3ef81 Compare October 9, 2025 18:15

jmarrero changed the title ~~WIP: Add --unified to use our container-storage for host images~~ WIP: Use our container-storage for host images if the refspec exists on it Oct 9, 2025

jmarrero changed the title ~~WIP: Use our container-storage for host images if the refspec exists on it~~ WIP: Use our container-storage for host images if the refspec exists Oct 9, 2025

jmarrero force-pushed the unified-storage-wip branch 2 times, most recently from e891e6c to be75055 Compare October 15, 2025 17:57

github-actions bot added the area/install Issues related to `bootc install` label Oct 15, 2025

jmarrero force-pushed the unified-storage-wip branch from 291fbd7 to fe10204 Compare November 11, 2025 19:53

jmarrero force-pushed the unified-storage-wip branch 3 times, most recently from 06be8fb to b801a17 Compare November 21, 2025 20:50

github-actions bot added the area/system-reinstall-bootc Issues related to system-reinstall-botoc label Nov 21, 2025

jmarrero force-pushed the unified-storage-wip branch from b801a17 to 68b1ba6 Compare November 21, 2025 20:56

jmarrero force-pushed the unified-storage-wip branch 3 times, most recently from ee300e3 to 50c4f70 Compare December 2, 2025 19:28

jmarrero changed the title ~~WIP: Use our container-storage for host images if the refspec exists~~ lib: Add experimental unified storage support for install Dec 2, 2025

gemini-code-assist bot reviewed Dec 2, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

crates/lib/src/install.rs Outdated Show resolved Hide resolved

crates/lib/src/deploy.rs Outdated Show resolved Hide resolved

cgwalters reviewed Dec 2, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

cgwalters reviewed Dec 4, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

jmarrero force-pushed the unified-storage-wip branch 4 times, most recently from 3803e4c to bb9883e Compare December 10, 2025 15:41

jmarrero force-pushed the unified-storage-wip branch from bb9883e to 04620ac Compare December 10, 2025 17:28

jmarrero marked this pull request as ready for review December 10, 2025 17:30

bootc-bot bot requested a review from ckyrouac December 10, 2025 17:30

gemini-code-assist bot reviewed Dec 10, 2025

View reviewed changes

crates/lib/src/deploy.rs Show resolved Hide resolved

crates/lib/src/deploy.rs Show resolved Hide resolved

crates/lib/src/deploy.rs Outdated Show resolved Hide resolved

jmarrero force-pushed the unified-storage-wip branch from 04620ac to 7651032 Compare December 10, 2025 17:44

gemini-code-assist bot reviewed Dec 10, 2025

View reviewed changes

crates/lib/src/image.rs Outdated Show resolved Hide resolved

cgwalters requested changes Dec 10, 2025

View reviewed changes

jmarrero force-pushed the unified-storage-wip branch from 7651032 to ddbba51 Compare December 11, 2025 03:41

gemini-code-assist bot reviewed Dec 11, 2025

View reviewed changes

jmarrero force-pushed the unified-storage-wip branch 2 times, most recently from 54ef798 to ba8e4ef Compare December 11, 2025 04:15

jmarrero force-pushed the unified-storage-wip branch from ba8e4ef to 7f11db8 Compare December 11, 2025 10:08

cgwalters mentioned this pull request Dec 11, 2025

Store container manifest #1830

Merged

		let sysroot = crate::cli::get_storage().await?;
		let ostree = sysroot.get_ostree()?;

lib: Add experimental unified storage support for install #1601

Are you sure you want to change the base?

lib: Add experimental unified storage support for install #1601

Conversation

jmarrero commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jlebon commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Oct 7, 2025

Uh oh!

jlebon commented Oct 7, 2025

Uh oh!

jmarrero commented Dec 2, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters commented Dec 2, 2025

Uh oh!

Uh oh!

Uh oh!

jmarrero commented Dec 10, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmarrero commented Dec 10, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jmarrero commented Sep 9, 2025 •

edited

Loading

jlebon commented Oct 7, 2025 •

edited

Loading