[Content Addressable] Ruby ABI hardening across indexes, APIs, events and admin

[jenshenny]

rubygems#6674

Follow-ups from the deep review of the content-addressable feature, one concern per commit:

Exclude content-addressable versions from the legacy Marshal indexes

Multiple Ruby ABI variants share a number and platform, so specs.4.8.gz / latest_specs.4.8.gz gained duplicate (name, number, platform) rows (worse in latest, where every variant is marked latest). Legacy clients can't install these gems anyway — the required_rubygems_version floor rejects them, and resolving the legacy entry constructs a platform-based download path while the file is stored content-addressed → 404 mid-install. ABI versions are now excluded from all three legacy index queries.

Expose ruby_abi in the version and rubygem payloads

API consumers and webhook receivers could not distinguish ABI variants — they appeared as identical entries differing only in sha. Adds ruby_abi to Version#payload (/api/v1/versions, the v2 version endpoint) and to Rubygem#payload, which feeds /api/v1/gems and the push/yank webhooks. Additive JSON/XML field.

Validate the ruby_abi format

The push path derives ruby_abi server-side and can only produce X.Y, but console sessions and backfills could persist arbitrary strings that would flow into the compact index and version identities. Same defense-in-depth as the sha256 and content_address format validations.

Allow resolving a specific Ruby ABI variant in the v2 API

Rubygem#find_public_version resolved by number and platform only, so the v2 version and contents endpoints returned an arbitrary variant when multiple ABI builds coexist (contents especially matters — variants have different files). Accepts a ruby_abi param, nil-scoped by default so existing lookups are unchanged, matching the deletions API semantics.

Record the Ruby ABI on the pushed version event

The yank/unyank/yank-forbidden events carry the ABI for auditing; the pushed event only had sha256 to distinguish variants. Records it there too for symmetric auditing.

Display the Ruby ABI for deleted versions in events and Avo

Event rows render to_title (already ABI-aware) when the version exists, but the fallback text for hard-deleted versions was number+platform only, making yanked ABI variants indistinguishable in the gem history. The fallback now includes the ABI, and the Avo Version resource surfaces ruby_abi and content_address (Deletion already shows its ABI).

Tophat

Pushes two ABI variants and a multi-ABI version through the real pipeline, then verifies every change in this PR: legacy index exclusion, v1/v2 payloads, v2 variant resolution, pushed events, and format validation.

Run from the repo root with the server on :3000 — paste the whole block into your console:

Tophat script (single paste)

bash <<'TOPHAT'
set -uo pipefail

BASE_URL="${BASE_URL:-http://localhost:3000}"
GEM="hardtophat$(date +%s)"
PLATFORM="x86_64-linux-musl"
BUILD_DIR=$(mktemp -d)
PASS=0 FAIL=0

step() { printf '\n\033[1m== %s\033[0m\n' "$*"; }
ok()   { PASS=$((PASS+1)); printf '  \033[32mPASS\033[0m %s\n' "$*"; }
bad()  { FAIL=$((FAIL+1)); printf '  \033[31mFAIL\033[0m %s\n' "$*"; }

runner() { bin/rails runner "$1" 2>/dev/null | grep "^OUT=" | cut -d= -f2-; }

step "Preflight: server on :3000"
curl -sf "$BASE_URL/" > /dev/null || { echo "Server not running — start it with ./.agents/skills/run-rubygems-org/smoke.sh"; exit 1; }
ok "server is up"

step "Setup: user + push API key + feature flag"
KEY=$(runner "
  user = User.find_or_create_by!(handle: \"hardtophat\") do |u|
    u.email = \"hardtophat@rubygems-test.org\"
    u.password = SecureRandom.hex(16)
    u.email_confirmed = true
  end
  FeatureFlag.enable_for_actor(FeatureFlag::CONTENT_ADDRESSABLE_GEM_PUSHES, user)
  raw = SecureRandom.hex(24)
  user.api_keys.create!(name: \"hard-#{SecureRandom.hex(4)}\", hashed_key: Digest::SHA256.hexdigest(raw), scopes: %w[push_rubygem])
  puts \"OUT=#{raw}\"")
[[ -n "$KEY" ]] && ok "ready" || { bad "setup failed — run bin/rails runner manually to see the error"; exit 1; }

step "Push: two ABI variants + one version supporting multiple Ruby ABIs"
ruby -rrubygems/package -e '
  gem_name, platform, build_dir = ARGV
  [["skinny32", "~> 3.2.0"], ["skinny34", "~> 3.4.0"], ["multi", ">= 3.2"]].each do |label, req|
    dir = File.join(build_dir, label); Dir.mkdir(dir)
    spec = Gem::Specification.new do |s|
      s.name = gem_name; s.version = "1.0.0"; s.platform = platform
      s.summary = "hardening tophat (#{label})"; s.authors = ["tophat"]; s.files = []
      s.required_ruby_version = req
    end
    Dir.chdir(dir) { Gem::Package.build(spec) }
  end' "$GEM" "$PLATFORM" "$BUILD_DIR" > /dev/null 2>&1
for f in skinny32 skinny34 multi; do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" -X POST "$BASE_URL/api/v1/gems" -H "Authorization: $KEY" \
              -H "Content-Type: application/octet-stream" --data-binary "@$BUILD_DIR/$f/$GEM-1.0.0-$PLATFORM.gem")
  [[ "$CODE" == "200" ]] && ok "pushed $f" || bad "push $f → $CODE"
done

step "Legacy Marshal indexes exclude the ABI variants"
ROWS=$(runner "
  r = Rubygem.find_by!(name: \"$GEM\")
  idx = Version.rows_for_index.select { |row| row[0] == r.name }
  latest = Version.rows_for_latest_index.select { |row| row[0] == r.name }
  puts \"OUT=#{idx.length},#{latest.length}\"")
[[ "$ROWS" == "1,1" ]] && ok "exactly one legacy row (the multi-ABI version) in specs + latest indexes" \
                       || bad "legacy index rows (specs,latest) = $ROWS, want 1,1"

step "ruby_abi exposed in the version and rubygem payloads"
V1=$(curl -s "$BASE_URL/api/v1/versions/$GEM.json")
echo "$V1" | ruby -rjson -e 'abis = JSON.parse(STDIN.read).map { |v| v["ruby_abi"] }.sort_by(&:to_s); exit(abis == [nil, "3.2", "3.4"].sort_by(&:to_s) ? 0 : 1)' \
  && ok "/api/v1/versions lists ruby_abi [nil, 3.2, 3.4]" || bad "/api/v1/versions payload: $(echo "$V1" | head -c 120)"
G1=$(curl -s "$BASE_URL/api/v1/gems/$GEM.json")
echo "$G1" | ruby -rjson -e 'exit(JSON.parse(STDIN.read).key?("ruby_abi") ? 0 : 1)' \
  && ok "/api/v1/gems payload (feeds webhooks) has the ruby_abi key" || bad "/api/v1/gems payload missing ruby_abi"

step "v2 API resolves a specific Ruby ABI variant"
for want in "3.2 200" "3.4 200" "3.3 404"; do
  abi=${want% *}; code=${want#* }
  GOT=$(curl -s -o /tmp/v2body -w "%{http_code}" "$BASE_URL/api/v2/rubygems/$GEM/versions/1.0.0.json?platform=$PLATFORM&ruby_abi=$abi")
  if [[ "$GOT" == "$code" ]]; then
    if [[ "$code" == "200" ]]; then
      ruby -rjson -e 'exit(JSON.parse(File.read("/tmp/v2body"))["ruby_abi"] == ARGV[0] ? 0 : 1)' "$abi" \
        && ok "v2 ruby_abi=$abi → 200 with matching payload" || bad "v2 ruby_abi=$abi payload mismatch"
    else
      ok "v2 ruby_abi=$abi → 404"
    fi
  else
    bad "v2 ruby_abi=$abi → $GOT, want $code"
  fi
done
GOT=$(curl -s "$BASE_URL/api/v2/rubygems/$GEM/versions/1.0.0.json?platform=$PLATFORM")
echo "$GOT" | ruby -rjson -e 'exit(JSON.parse(STDIN.read)["ruby_abi"].nil? ? 0 : 1)' \
  && ok "v2 without ruby_abi resolves the multi-ABI version" || bad "v2 default resolution: $(echo "$GOT" | head -c 100)"

step "Pushed events carry the Ruby ABI"
EV=$(runner "
  r = Rubygem.find_by!(name: \"$GEM\")
  abis = r.events.where(tag: Events::RubygemEvent::VERSION_PUSHED).map { |e| e.additional.ruby_abi }.sort_by(&:to_s)
  puts \"OUT=#{abis.inspect}\"")
[[ "$EV" == '[nil, "3.2", "3.4"]' ]] && ok "three VERSION_PUSHED events with ruby_abi [nil, 3.2, 3.4]" \
                                     || bad "pushed event abis: $EV"

step "ruby_abi format is validated"
VAL=$(runner "
  v = Version.new(ruby_abi: \"banana\")
  v.valid?
  puts \"OUT=#{v.errors[:ruby_abi].first}\"")
[[ "$VAL" == "is invalid" ]] && ok "ruby_abi: \"banana\" is rejected" || bad "format validation: $VAL"

rm -rf "$BUILD_DIR"
printf '\n\033[1m%d passed, %d failed\033[0m\n' "$PASS" "$FAIL"
echo "Manual check: /admin/resources/versions shows Ruby ABI + content address columns for $GEM"
[[ $FAIL -eq 0 ]]
TOPHAT

Output (14 passed, 0 failed)

== Preflight: server on :3000
  PASS server is up
== Setup: user + push API key + feature flag
  PASS ready
== Push: two ABI variants + one version supporting multiple Ruby ABIs
  PASS pushed skinny32 / skinny34 / multi
== Legacy Marshal indexes exclude the ABI variants
  PASS exactly one legacy row (the multi-ABI version) in specs + latest indexes
== ruby_abi exposed in the version and rubygem payloads
  PASS /api/v1/versions lists ruby_abi [nil, 3.2, 3.4]
  PASS /api/v1/gems payload (feeds webhooks) has the ruby_abi key
== v2 API resolves a specific Ruby ABI variant
  PASS v2 ruby_abi=3.2 → 200 with matching payload
  PASS v2 ruby_abi=3.4 → 200 with matching payload
  PASS v2 ruby_abi=3.3 → 404
  PASS v2 without ruby_abi resolves the multi-ABI version
== Pushed events carry the Ruby ABI
  PASS three VERSION_PUSHED events with ruby_abi [nil, 3.2, 3.4]
== ruby_abi format is validated
  PASS ruby_abi: "banana" is rejected
14 passed, 0 failed

Avo display is a manual check: /admin/resources/versions shows the Ruby ABI and content address columns for the tophat gem's versions.