RFC: Use a Gemfile for Spinel projects (for now)

[OriPekelman]

Summary

Spinel-compiled projects have no shared way to declare or exchange dependencies,
so each one vendors by hand. This RFC proposes a deliberately small, temporary
answer that requires no new design:

1. Declare dependencies in a standard Gemfile. Mark the project with
   ruby "3.x", engine: "spinel", engine_version: "0.0.0". That's the whole
   convention. It buys a familiar manifest, path:/git: sources for sibling
   projects, and Bundler's resolver + lockfile — for free.

2. A small Bundler plugin (spinelgems) that does two
   things: (a) makes it work — places resolved dependencies where Spinel can
   actually find them; and (b) gates — flags gems Spinel can't compile early,
   so the experience is nicer than a silent miscompile.

The point is to postpone the big discussion. Spinel isn't near a release, and
there's no expectation it adopts any particular dependency-management design. But
projects need some interop today — so let's borrow a format everyone knows and
revisit later, rather than design a package manager now.

Motivation

Interop is already happening, ad-hoc. Roundhouse (a separate author's project)
vendors part of Tep; our own projects carry a mix of rsync'd copies and hand-
maintained concatenation. Every project reinvents "get this other code into a
shape Spinel will compile." A shared, boring convention removes that duplication
without anyone committing to a long-term model.

Two facts make a thin tool worthwhile (both verified — Bundler 2.7.2 / CRuby):

• Bundler already does the right thing with the engine marker. bundle lock
  ignores engine: "spinel" and resolves normally (exit 0); only bundle install fires the guard ("Your Ruby engine is ruby, but your Gemfile specified
  spinel", exit 18). So the convention costs nothing and fails loudly in the
  right place.

• Spinel doesn't fail loudly on unsupported Ruby. It prints
  warning: … cannot resolve call to 'eval' … (emitting 0) and degrades the call
  to a no-op, exiting 0; some constructs (and silent miscompiles) warn not at
  all. So "it compiled" ≠ "it works" — which is why a little gating help is worth
  having.

Proposal 1 — the Gemfile convention

A Spinel project's Gemfile:

source "https://rubygems.org"
ruby "3.3.0", engine: "spinel", engine_version: "0.0.0"

gem "tep", git: "https://…/tep.git"   # sibling projects via path:/git:
gem "some_pure_ruby_lib"              # third-party, if it compiles

Nothing here is novel — that's the point. bundle lock produces a normal
lockfile; siblings resolve through path:/git: sources (replacing rsync /
manual vendoring); third-party gems resolve from rubygems.org. The engine marker
documents the target and guards bundle install.

Proposal 2 — the Bundler plugin

(a) Make it work — placement

Spinel has no load path (plain require "x" resolves only against
<spinel>/lib) and inlines require_relative. So a resolved dependency has to
be placed where Spinel will follow it. The plugin vendors each locked gem's
lib/ into a project dir and generates a require_relative manifest:

spinel-compat vendor            # reads Gemfile.lock
# -> vendor/spinel/<gem>/lib/…  and  vendor/spinel/deps.rb

A Spinel program then just require_relative "vendor/spinel/deps". This is the
reusable form of what projects already do by hand (concatenation scripts, partial
vendoring).

(b) Gating — a nicer experience

Because Spinel silently no-ops unsupported Ruby, the plugin probes gems (compile

• static scan, optionally a differential CRuby-vs-Spinel run) and flags ones that
  won't work — at bundle lock time, with reasons that name the missing feature:

bundle spinel-lock     # bundle lock, then report incompatible gems

Verdicts are forward-compatible: each is keyed on the Spinel revision (Spinel
ships no version, so we key on the git rev / binary hash, scoped by platform).
Upgrade Spinel → re-probe → a gem rejected today clears the moment the feature it
needed lands. No hand-maintained blocklist.

Gating is advisory by design — placement and compatibility are different jobs.
The plugin makes the failure visible and early; it doesn't try to be a wall.

Open questions (for discussion, not asks)

These are things we ran into, offered as discussion points — not feature
requests, and not assuming Spinel wants any of them:

• Signalling unsupported constructs. Today they're silent no-ops at exit 0.
  Would a non-zero exit or a structured (e.g. JSON) diagnostic — perhaps behind a
  --check/--strict flag — be in scope or interest? It's the difference
  between inferring compatibility from stderr and reading it from a contract.
• A stable engine identity. We key verdicts on a git rev / binary hash for
  now. If Spinel ever wanted to expose a version/rev, the ledger and the
  engine_version: marker could line up — but there's no rush.
• Resolving require beyond <spinel>/lib. A load-path notion would let
  multi-file gems and stdlib shims resolve (and would make probing more
  accurate). Possibly useful well beyond this tool; possibly out of scope.

None of these are needed for the convention or the plugin to be useful today.
They're just where the seams are, if there's ever appetite to smooth them.

[OriPekelman]

cc @neidiom @matz around #747

[neidiom]

@OriPekelman I like it. Makes sense. I lean towards Proposal 1 — the Gemfile convention.

[rubys]

nice

[matz]

RFC / discussion thread — closing as not a bug. Future Gemfile / dependency-management work can land via a fresh issue or PR.

[matz]

Reopened — discussion / RFC threads should stay open until the maintainer weighs in. (Apologies for the premature close.)

[neidiom]

@OriPekelman what is the next step?

[OriPekelman]

@matz @neidiom — in practice this needs no coordination on Spinel's side. I am already using this on my Spinel projects.

See:

• https://github.com/OriPekelman/toy/blob/main/Gemfile
• https://github.com/OriPekelman/toy/blob/main/examples/04_serve_http.rb#L24

You can keep the language's scope intact and treat Gemfiles as the de-facto convention; my Bundler plugin handles placement (where Spinel will find code — no load path → require_relative manifest) and gates incompatibility at lock-time:

https://github.com/OriPekelman/spinelgems/

I'm also working through the ecosystem: a wholesale survey over the ~193k indexed gems on rubygems.org, classifying each at the current Spinel rev. For rejected gems it captures which feature was the blocker — the cannot resolve call to 'X' signal is gold; it's a prioritized roadmap of core/stdlib calls Spinel could learn next (given Spinel's current scope, no metaprogramming). C-extension gems are handled via a spinel-ext.json manifest the supplier ships; vendor compiles the .c → .o and substitutes the placeholder. This is for "spinel aware gems" (declared via a spinel-ext.json manifest the supplier ships; the survey marks vanilla CRuby c-ext gems rejected:c-extension since Spinel can't load .sos — that's just the cost of AOT). I think further study is necessary on those.

This already allowed me to identify "low hanging fruit" that adds a bunch of Gems to the "supported column": PR #1010 (merged) — fixed the namespaced-constant regex blocker the survey surfaced. The current 193k run has ~8h to go; I'll share full stats + the candidate-call ranking when it completes. Catalog: https://oripekelman.github.io/spinelgems/ (currently a snapshot at git:397115c; the fresh-upstream verdicts — 1 behavior-verified + 235 loaded — land in the next few hours).

I also built a rubygems.org proxy that we could run on spinelgems.org and that could filter-in only the supported Gems (as a backstop measure - imagining that if any of this is useful it will not be needed in the future as Gems will remain Gems.) But I am unsure about its actual usefulness. If I decide to start importing actual ruby gems into any of my spinel projects I might take it live. See: https://github.com/OriPekelman/spinelgems/blob/main/lib/bundler/spinel/proxy.rb - If it is useful to you - I can just spin-it up.

[OriPekelman]

I am wary of community driven potentially future-de-facto standards so I will try and move my spinel-ext.json to the "consumer side" (the bundler plugin) rather than assuming "Spinel Gems" authors do anything special and try to auto-detect ffi_cflags declarations... or something to that order. So coordination cost remains at 0.

[OriPekelman]

Survey update — partial run at the post-#1011 master rev

Running it on my server; switched from a thread-pool to process sharding mid-stream (the GIL was bottlenecking on a single core regardless of --jobs, so wall-clock was network-bound). The partial — 84,324 gems @ git:a03bb49 — is now live as a browsable catalog: https://oripekelman.github.io/spinelgems/catalog.html

Verdict mix (44% through the 193k index)

clean ✓	risky ~	rejected ✗
11,520 (13.7%)	7,478 (8.9%)	65,326 (77.5%)

Blockers, ranked by occurrences (cross-gem)

bucket	count	nature
candidate core/stdlib calls	250,859	roadmap signal — calls Spinel could learn
metaprogramming / reflection	80,333	out of scope by design
no load path (require / needs:)	370,030	probe artifact — sees cross-file calls as "unresolved" when the defining require wasn't followed; most aren't real blockers
analyzer failed / timed out	13,779	compiler hardening (per-gem timeout / analyzer crash)
C extensions	1,708	uncompilable under Spinel (no dlopen); Spinel-targeted gems handled via ffi_cflags

Top candidate calls

new 13,893 · [] 10,044 · expand_path 3,576 · parse 3,309 · include? 3,146 · unshift 2,920 · join 2,704 · empty? 2,676 · extend 2,188 · include 2,099 · first 2,052 · split 1,848 · call 1,835 · length 1,829 · delete 1,640 · gsub 1,273 · open 1,217 · keys 1,155 · glob 1,142 · read 1,091

Full ranked list: candidates.tsv.

Side notes

• Compile-time fold File.expand_path(rel, __dir__) so top-level ffi_cflags can be self-referential #1011 is wired into vendor — spinel-compat detect-ext now recognizes both ffi_cflags "@PLACEHOLDER@" (legacy) and ffi_cflags File.expand_path("X.o", __dir__) (post-Compile-time fold File.expand_path(rel, __dir__) so top-level ffi_cflags can be self-referential #1011 const-fold), and spinel-compat vendor places the .o at the right path for either form. Verified end-to-end. Authors don't need to do or know anything — full handling stays consumer-side.
• Resumed the survey on a process-sharded runner (measured ~18 gems/sec now vs ~3 before); should land in the next couple of hours. Tomorrow I'll re-run against fresh master so the catalog tracks any of today's movement. Will post again when the full corpus is in.