[Content Addressable] Bundler Local Cache and Lockfile - #178
Open
OughtPuts wants to merge 10 commits into
Open
Conversation
Allow `gem build --ruby-abi X.Y` to build a gem that only supports a single Ruby ABI (a skinny gem) with a content addressable file name of the form <name>-<version>-<sha8>.gem, where the suffix is derived from the SHA-256 digest of the gem contents. The gem is built in memory and validated against the requested Ruby ABI. If required_ruby_version is unset it is derived from the ABI; the passed-in spec is only updated once the build succeeds. Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
Co-authored-by: Jenny Shen <jenny.shen@shopify.com>
- Parse compact-index suffix as content address when hex + platform:= requirement present; carry content_address through resolver. - Prefer content-addressed gems in resolver when platform specificity ties. - Download and install SHA-named gem archives; fetch SHA-named gemspecs for development dependencies. - Preserve distinct resolver candidates by content_address in ==/hash. - Update BasicSpecification.content_address? calls to Gem::ContentAddress.match? (PR #172 refactor). Assisted-By: devx/fff388fa-6f3f-4c7a-8385-6e602f16830e
Assisted-By: devx/5f8c508e-3e50-4812-b2c3-84379e7c3fce
Allow gem yank to select CA gems by Ruby ABI
There was a problem hiding this comment.
Pull request overview
This PR adds Bundler/RubyGems support for content-addressable gem artifacts end-to-end, including compact-index v2 test infrastructure, lockfile encoding/parsing, and ensuring local cache resolution works with content-addressed filenames.
Changes:
- Extend specification/name tuple objects to carry
content_address, and update resolution & lockfile serialization/parsing accordingly. - Add Artifice compact-index v2 helpers to model content-addressable gems in specs, including platform metadata injection.
- Add/adjust specs to cover lockfile formatting/parsing and local-cache round-trips for content-addressable gems.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| spec/support/builders.rb | Build helper now supports producing content-addressable gem files (Ruby ABI–scoped). |
| spec/support/artifice/helpers/compact_index.rb | Checksum calculation updated to use spec.full_name gem filenames. |
| spec/support/artifice/helpers/compact_index_v2.rb | New compact-index v2 Artifice helper with content-addressable gem support and metadata injection. |
| spec/support/artifice/helpers/compact_index_cooldown.rb | Removed (superseded by v2 helper). |
| spec/support/artifice/compact_index_v2.rb | New Artifice activation entrypoint for the v2 helper. |
| spec/support/artifice/compact_index_cooldown.rb | Removed (tests now use compact_index_v2). |
| spec/other/ext_spec.rb | Add coverage for Gem::NameTuple#lock_name and Bundler::LazySpecification#to_lock with content addresses. |
| spec/install/gemfile/content_addressable_spec.rb | New integration specs for content-addressable install, caching, and lockfile round-trips. |
| spec/install/cooldown_spec.rb | Switch cooldown tests to use the compact-index v2 Artifice setup. |
| spec/bundler/remote_specification_spec.rb | Rename “platform” to “suffix” in test setup and add content-address expectation coverage. |
| spec/bundler/override_spec.rb | Ensure overrides stubs include content_address. |
| spec/bundler/lockfile_parser_spec.rb | Add parsing coverage for lockfile lines with a trailing content address token. |
| spec/bundler/endpoint_specification_spec.rb | Add coverage for CA suffix handling via platform metadata and remote fetching behavior. |
| lib/rubygems/specification.rb | Include content_address in Specification#name_tuple. |
| lib/rubygems/safe_marshal.rb | Permit Gem::NameTuple to safely marshal/unmarshal @content_address. |
| lib/rubygems/name_tuple.rb | Add content_address field on Gem::NameTuple. |
| lib/bundler/stub_specification.rb | Propagate content_address when building StubSpecifications. |
| lib/bundler/source/rubygems.rb | Preserve content address when swapping remote specs after download. |
| lib/bundler/rubygems_integration.rb | Populate spec content_address from Gem::Package when available. |
| lib/bundler/rubygems_gem_installer.rb | Call RubyGems’ content-address assignment hook when supported. |
| lib/bundler/rubygems_ext.rb | Compatibility shims for older RubyGems plus lock name behavior for content-addressed tuples. |
| lib/bundler/resolver.rb | De-dupe resolved specs including content_address to avoid collapsing distinct CA variants. |
| lib/bundler/remote_specification.rb | Add content_address and include it in full_name when applicable. |
| lib/bundler/match_platform.rb | Prefer compatible content-addressable candidates when available. |
| lib/bundler/lockfile_parser.rb | Parse trailing content-address token on spec lines and propagate into LazySpecification. |
| lib/bundler/lazy_specification.rb | Track content_address, include it in full_name, and emit it in lock output. |
| lib/bundler/fetcher.rb | Treat the 3rd token as a “suffix” and pass through to endpoint/remote specification constructors. |
| lib/bundler/endpoint_specification.rb | Add CA suffix parsing via metadata-provided platform and fetch remote specs using CA suffix when present. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Assisted-By: devx/ce8ef461-c04c-4d00-8751-1b2488fe8a4b
Assisted-By: devx/ce8ef461-c04c-4d00-8751-1b2488fe8a4b
Assisted-By: devx/ce8ef461-c04c-4d00-8751-1b2488fe8a4b
Co-authored-by: Harriet Oughton <harriet.oughton@shopify.com>
OughtPuts
force-pushed
the
ho/bundler-local-cache-lockfiles
branch
3 times, most recently
from
August 19, 2026 09:25
474c200 to
e884790
Compare
OughtPuts
force-pushed
the
ho/bundler-local-cache-lockfiles
branch
from
August 19, 2026 09:30
e884790 to
9f26cc0
Compare
OughtPuts
force-pushed
the
ho/bundler-local-cache-lockfiles
branch
from
August 19, 2026 09:46
9f26cc0 to
b985f74
Compare
OughtPuts
force-pushed
the
ho/bundler-local-cache-lockfiles
branch
from
August 19, 2026 09:58
b985f74 to
7179457
Compare
OughtPuts
marked this pull request as ready for review
August 19, 2026 12:29
OughtPuts
commented
Aug 19, 2026
| spec = @specs[full_name] | ||
| lock_name = Gem::NameTuple.new(name, version, platform).lock_name | ||
| @specs_by_lock_name ||= @specs.values.to_h {|candidate| [candidate.lock_name, candidate] } | ||
| spec = @specs[full_name] || @specs_by_lock_name[lock_name] |
Author
There was a problem hiding this comment.
This allows us to correctly find the CA gem spec by either the full_name OR the lock_name after the recent naming changes, when storing the checksum.
OughtPuts
force-pushed
the
ho/bundler-local-cache-lockfiles
branch
from
August 19, 2026 12:49
7179457 to
582815e
Compare
OughtPuts
force-pushed
the
feature-branch-ca-changes-rubygems
branch
from
August 19, 2026 15:56
aad98a3 to
0caed62
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ruby#9733
TL;DR
This PR updates the bundler install flow for locally cached gems to support content addressable gem naming.
Description
This PR builds on the Bundler remote-install support for content-addressed gems.
It writes the gem’s content address and platform to
Gemfile.lock, then parses those values back intoLazySpecificationduring a subsequent install. This lets Bundler identify the content-addressed.geminvendor/cacheand install it withbundle install --local, without contacting the remote source again.The same content-addressed name is used for the installed gem directory in both remote and local installation paths, ensuring lockfile round-trips produce a consistent on-disk layout.
Tests
Updates made to
lockfile_parser_spec,content_addressable_specandext_specto cover the changes.Tophatting
bundle installfrom the directory that contains your gemfile, using a Ruby version compatible with the content addressable gem.Gemfile.lockcontains a content address:/cache/directory)bundle install --local. Confirm the content addressable gem installs correctly still and that it uses the correct content addressable name.I've performed this tophat locally and it's all looking good! ✅