Skip to content

[Content Addressable] Bundler Local Cache and Lockfile - #178

Open
OughtPuts wants to merge 10 commits into
feature-branch-ca-changes-rubygemsfrom
ho/bundler-local-cache-lockfiles
Open

[Content Addressable] Bundler Local Cache and Lockfile#178
OughtPuts wants to merge 10 commits into
feature-branch-ca-changes-rubygemsfrom
ho/bundler-local-cache-lockfiles

Conversation

@OughtPuts

@OughtPuts OughtPuts commented Aug 18, 2026

Copy link
Copy Markdown

ruby#9733

TL;DR

This PR updates the bundler install flow for locally cached gems to support content addressable gem naming.

Description

This PR builds on the Bundler remote-install support for content-addressed gems.

It writes the gem’s content address and platform to Gemfile.lock, then parses those values back into LazySpecification during a subsequent install. This lets Bundler identify the content-addressed .gem in vendor/cache and install it with bundle install --local, without contacting the remote source again.

The same content-addressed name is used for the installed gem directory in both remote and local installation paths, ensuring lockfile round-trips produce a consistent on-disk layout.

Tests

Updates made to lockfile_parser_spec, content_addressable_spec and ext_spec to cover the changes.

Tophatting

  1. Start a local rubygems.org server, that contains a content-addressable gem.
  2. Set your gemfile to point to the local server and a content-addressable gem that is available via that server.
  3. Run bundle install from the directory that contains your gemfile, using a Ruby version compatible with the content addressable gem.
  4. Confirm that the Gemfile.lock contains a content address:
GEM
  remote: http://localhost:3000/
  specs:
    ca_test_harriet (0.3.5-arm64-darwin-23) d26b82f6

PLATFORMS
  arm64-darwin-23

DEPENDENCIES
  ca_test_harriet (= 0.3.5)

CHECKSUMS
  ca_test_harriet (0.3.5-arm64-darwin-23)

BUNDLED WITH
  4.1.0.dev
  1. Remove any locally installed gem files other than the cached files (normally in the /cache/ directory)
  2. Re-run bundle install --local. Confirm the content addressable gem installs correctly still and that it uses the correct content addressable name.

I've performed this tophat locally and it's all looking good! ✅

OughtPuts and others added 5 commits August 7, 2026 13:36
Allow `gem build --ruby-abi X.Y` to build a gem that only supports a
single Ruby ABI (a skinny gem) with a content addressable file name of
the form <name>-<version>-<sha8>.gem, where the suffix is derived from
the SHA-256 digest of the gem contents.

The gem is built in memory and validated against the requested Ruby
ABI. If required_ruby_version is unset it is derived from the ABI; the
passed-in spec is only updated once the build succeeds.

Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
- Parse compact-index suffix as content address when hex + platform:=
  requirement present; carry content_address through resolver.
- Prefer content-addressed gems in resolver when platform specificity ties.
- Download and install SHA-named gem archives; fetch SHA-named gemspecs
  for development dependencies.
- Preserve distinct resolver candidates by content_address in ==/hash.
- Update BasicSpecification.content_address? calls to
  Gem::ContentAddress.match? (PR #172 refactor).

Assisted-By: devx/fff388fa-6f3f-4c7a-8385-6e602f16830e
Assisted-By: devx/5f8c508e-3e50-4812-b2c3-84379e7c3fce
Allow gem yank to select CA gems by Ruby ABI
@OughtPuts
OughtPuts marked this pull request as draft August 18, 2026 17:50
@OughtPuts
OughtPuts requested a lite review from Copilot August 18, 2026 17:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds Bundler/RubyGems support for content-addressable gem artifacts end-to-end, including compact-index v2 test infrastructure, lockfile encoding/parsing, and ensuring local cache resolution works with content-addressed filenames.

Changes:

  • Extend specification/name tuple objects to carry content_address, and update resolution & lockfile serialization/parsing accordingly.
  • Add Artifice compact-index v2 helpers to model content-addressable gems in specs, including platform metadata injection.
  • Add/adjust specs to cover lockfile formatting/parsing and local-cache round-trips for content-addressable gems.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
spec/support/builders.rb Build helper now supports producing content-addressable gem files (Ruby ABI–scoped).
spec/support/artifice/helpers/compact_index.rb Checksum calculation updated to use spec.full_name gem filenames.
spec/support/artifice/helpers/compact_index_v2.rb New compact-index v2 Artifice helper with content-addressable gem support and metadata injection.
spec/support/artifice/helpers/compact_index_cooldown.rb Removed (superseded by v2 helper).
spec/support/artifice/compact_index_v2.rb New Artifice activation entrypoint for the v2 helper.
spec/support/artifice/compact_index_cooldown.rb Removed (tests now use compact_index_v2).
spec/other/ext_spec.rb Add coverage for Gem::NameTuple#lock_name and Bundler::LazySpecification#to_lock with content addresses.
spec/install/gemfile/content_addressable_spec.rb New integration specs for content-addressable install, caching, and lockfile round-trips.
spec/install/cooldown_spec.rb Switch cooldown tests to use the compact-index v2 Artifice setup.
spec/bundler/remote_specification_spec.rb Rename “platform” to “suffix” in test setup and add content-address expectation coverage.
spec/bundler/override_spec.rb Ensure overrides stubs include content_address.
spec/bundler/lockfile_parser_spec.rb Add parsing coverage for lockfile lines with a trailing content address token.
spec/bundler/endpoint_specification_spec.rb Add coverage for CA suffix handling via platform metadata and remote fetching behavior.
lib/rubygems/specification.rb Include content_address in Specification#name_tuple.
lib/rubygems/safe_marshal.rb Permit Gem::NameTuple to safely marshal/unmarshal @content_address.
lib/rubygems/name_tuple.rb Add content_address field on Gem::NameTuple.
lib/bundler/stub_specification.rb Propagate content_address when building StubSpecifications.
lib/bundler/source/rubygems.rb Preserve content address when swapping remote specs after download.
lib/bundler/rubygems_integration.rb Populate spec content_address from Gem::Package when available.
lib/bundler/rubygems_gem_installer.rb Call RubyGems’ content-address assignment hook when supported.
lib/bundler/rubygems_ext.rb Compatibility shims for older RubyGems plus lock name behavior for content-addressed tuples.
lib/bundler/resolver.rb De-dupe resolved specs including content_address to avoid collapsing distinct CA variants.
lib/bundler/remote_specification.rb Add content_address and include it in full_name when applicable.
lib/bundler/match_platform.rb Prefer compatible content-addressable candidates when available.
lib/bundler/lockfile_parser.rb Parse trailing content-address token on spec lines and propagate into LazySpecification.
lib/bundler/lazy_specification.rb Track content_address, include it in full_name, and emit it in lock output.
lib/bundler/fetcher.rb Treat the 3rd token as a “suffix” and pass through to endpoint/remote specification constructors.
lib/bundler/endpoint_specification.rb Add CA suffix parsing via metadata-provided platform and fetch remote specs using CA suffix when present.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread lib/bundler/lockfile_parser.rb
girachawda and others added 4 commits August 18, 2026 15:27
Assisted-By: devx/ce8ef461-c04c-4d00-8751-1b2488fe8a4b
Assisted-By: devx/ce8ef461-c04c-4d00-8751-1b2488fe8a4b
Assisted-By: devx/ce8ef461-c04c-4d00-8751-1b2488fe8a4b
Co-authored-by: Harriet Oughton <harriet.oughton@shopify.com>
@OughtPuts
OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch 3 times, most recently from 474c200 to e884790 Compare August 19, 2026 09:25
@OughtPuts
OughtPuts requested a lite review from Copilot August 19, 2026 09:26
@OughtPuts
OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from e884790 to 9f26cc0 Compare August 19, 2026 09:30

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Comment thread lib/bundler/lockfile_parser.rb Outdated
Comment thread lib/bundler/stub_specification.rb
Comment thread lib/bundler/rubygems_integration.rb
Comment thread lib/bundler/rubygems_gem_installer.rb Outdated
Comment thread lib/bundler/lazy_specification.rb Outdated
@OughtPuts
OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 9f26cc0 to b985f74 Compare August 19, 2026 09:46
@OughtPuts
OughtPuts requested a lite review from Copilot August 19, 2026 09:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comment thread spec/install/gemfile/content_addressable_spec.rb Outdated
@OughtPuts
OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from b985f74 to 7179457 Compare August 19, 2026 09:58
@OughtPuts
OughtPuts requested a lite review from Copilot August 19, 2026 09:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Comment thread lib/bundler/lockfile_parser.rb
Comment thread lib/bundler/lockfile_parser.rb
@OughtPuts
OughtPuts marked this pull request as ready for review August 19, 2026 12:29
spec = @specs[full_name]
lock_name = Gem::NameTuple.new(name, version, platform).lock_name
@specs_by_lock_name ||= @specs.values.to_h {|candidate| [candidate.lock_name, candidate] }
spec = @specs[full_name] || @specs_by_lock_name[lock_name]

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This allows us to correctly find the CA gem spec by either the full_name OR the lock_name after the recent naming changes, when storing the checksum.

@OughtPuts
OughtPuts force-pushed the ho/bundler-local-cache-lockfiles branch from 7179457 to 582815e Compare August 19, 2026 12:49
@OughtPuts
OughtPuts requested a lite review from Copilot August 19, 2026 12:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants