Skip to content

Commit acca347

Browse files
feat(readme): ADR-004 Phase B — readme-derive reusable workflow + vocab gate (#475)
## What Implements the **accepted ADR-004** (README single-source derivation). Phase B: the machinery, no repo flips yet. - **`readme-derive-reusable.yml`** — `workflow_call` reusable. Reads a `[publishing.readme]` table from descriptiles `ECOSYSTEM.a2ml` (6a2 fallback during migration). For opted-in repos only: `asciidoctor-reducer` → docbook → `pandoc` gfm → GENERATED+SPDX stamp → vocab gate → **fail-and-tell** freshness check. Undeclared repos are skipped entirely — the anti-runaway guard. - **`.github/scripts/readme-vocab-gate.sh`** — fail-closed ≥98% vocabulary-preservation gate. Tested locally: 100% on a faithful conversion, hard-fails on real content loss. - **`doc-format.yml`** — reverses the June-runaway-era rule. `README.adoc` is canonical estate-wide; a `README.md` beside it is valid **only** as a GENERATED-stamped derived artefact, else flagged as a hand-maintained duplicate. A lone `README.adoc` is correct (no more nudge-to-Markdown). - **`ECOSYSTEM.a2ml.template`** — documented optional `[publishing.readme]` stanza. - **ADR-004** — status → Accepted; the three §Open items resolved at their proposed defaults (ECOSYSTEM.a2ml block, 98% gate, fail-and-tell); illustrative sketch aligned to the real flat-TOML ECOSYSTEM format. ## Accepted defaults (no owner veto on §Open items) 1. Declaration in `ECOSYSTEM.a2ml` `[publishing.readme]` (not standalone PUBLISHING.a2ml) 2. Vocab gate floor: 98% 3. Freshness: fail-and-tell (no self-commit) ## Next (Phase C) Pilot flips of **boj-server** (consumer=glama) and **hyperpolymath/hyperpolymath** (consumer=github-profile) to authored `.adoc` + derived `.md` — those PRs will be left un-armed pending visual verification on Glama / the profile page, per the ADR Phase C gate. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 68cadb2 commit acca347

5 files changed

Lines changed: 330 additions & 31 deletions

File tree

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
#!/usr/bin/env bash
2+
# SPDX-License-Identifier: MPL-2.0
3+
#
4+
# readme-vocab-gate.sh — fail-closed vocabulary-preservation gate for the
5+
# README single-source derivation (ADR-004).
6+
#
7+
# Guards against content loss when README.adoc is converted to README.md.
8+
# Extracts the prose vocabulary (lowercased, >=4-char alphanumeric tokens,
9+
# minus a markup stoplist) from BOTH the canonical .adoc and the derived .md,
10+
# then requires the derived file to retain at least THRESHOLD% of the
11+
# canonical file's unique content tokens. Exits non-zero if it falls short,
12+
# so a lossy conversion can never land silently.
13+
#
14+
# Usage: readme-vocab-gate.sh <canonical.adoc> <derived.md> [threshold_pct]
15+
# threshold_pct defaults to 98.
16+
set -euo pipefail
17+
18+
CANON="${1:?usage: readme-vocab-gate.sh <canonical.adoc> <derived.md> [threshold_pct]}"
19+
DERIVED="${2:?missing derived file}"
20+
THRESHOLD="${3:-98}"
21+
22+
[ -f "$CANON" ] || { echo "::error::vocab-gate: canonical file not found: $CANON"; exit 2; }
23+
[ -f "$DERIVED" ] || { echo "::error::vocab-gate: derived file not found: $DERIVED"; exit 2; }
24+
25+
# Markup / syntax tokens that appear in one format but not the other and carry
26+
# no prose meaning. Kept deliberately small; extend only with true noise.
27+
STOPLIST='http|https|www|html|adoc|markdown|xmlns|docbook|svg|png|jpg|jpeg|gif|href|link|image|images|toc|sectnums|sectnumlevels|toclevels|revnumber|revdate|revremark|source|highlighter|rouge|icons|font|doctype|imagesdir|experimental|nbsp|middot|span|div|code|pre|nolang|lang'
28+
29+
# Extract unique content tokens from a file.
30+
# - lowercase
31+
# - split on any non-alphanumeric
32+
# - keep tokens with length >= 4
33+
# - drop pure numbers and stoplist tokens
34+
extract_tokens() {
35+
tr '[:upper:]' '[:lower:]' < "$1" \
36+
| tr -c 'a-z0-9' '\n' \
37+
| awk 'length($0) >= 4 && $0 !~ /^[0-9]+$/' \
38+
| grep -Evx "$STOPLIST" \
39+
| sort -u
40+
}
41+
42+
CANON_TOKENS="$(mktemp)"
43+
DERIVED_TOKENS="$(mktemp)"
44+
trap 'rm -f "$CANON_TOKENS" "$DERIVED_TOKENS"' EXIT
45+
46+
extract_tokens "$CANON" > "$CANON_TOKENS"
47+
extract_tokens "$DERIVED" > "$DERIVED_TOKENS"
48+
49+
TOTAL=$(wc -l < "$CANON_TOKENS" | tr -d ' ')
50+
if [ "$TOTAL" -eq 0 ]; then
51+
echo "::error::vocab-gate: canonical file has no content tokens — refusing to pass a vacuous gate"
52+
exit 3
53+
fi
54+
55+
# Tokens present in canonical but MISSING from derived.
56+
MISSING="$(comm -23 "$CANON_TOKENS" "$DERIVED_TOKENS")"
57+
MISSING_COUNT=$(printf '%s\n' "$MISSING" | grep -c . || true)
58+
KEPT=$(( TOTAL - MISSING_COUNT ))
59+
60+
# Integer percentage with one-decimal reporting via *10 arithmetic.
61+
COVERAGE_X10=$(( KEPT * 1000 / TOTAL ))
62+
THRESHOLD_X10=$(( THRESHOLD * 10 ))
63+
64+
printf 'vocab-gate: %d/%d canonical content tokens retained (%d.%d%%), threshold %d%%\n' \
65+
"$KEPT" "$TOTAL" "$(( COVERAGE_X10 / 10 ))" "$(( COVERAGE_X10 % 10 ))" "$THRESHOLD"
66+
67+
if [ "$COVERAGE_X10" -lt "$THRESHOLD_X10" ]; then
68+
echo "::error::vocab-gate FAILED — derived README dropped ${MISSING_COUNT} content token(s) below the ${THRESHOLD}% floor."
69+
echo "First missing tokens:"
70+
printf '%s\n' "$MISSING" | head -40 | sed 's/^/ - /'
71+
exit 1
72+
fi
73+
74+
echo "✓ vocab-gate passed"

.github/workflows/doc-format.yml

Lines changed: 21 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -40,12 +40,19 @@ jobs:
4040
fi
4141
done
4242
43-
# README MUST be Markdown: it renders in GitHub community-health, the GitHub
44-
# profile, and external MCP directories (Glama) — all of which show AsciiDoc
45-
# as raw markup. So README.md is canonical; README.adoc is a duplicate to remove.
43+
# README: AsciiDoc is canonical estate-wide (ADR-004). A README.md
44+
# alongside README.adoc is legitimate ONLY when it is a *derived*
45+
# artefact — it must carry the GENERATED banner and the repo must
46+
# declare a [publishing.readme] consumer in descriptiles. A hand-
47+
# maintained README.md next to README.adoc is the duplicate to remove
48+
# (this is the case the June 2026 runaway inverted).
4649
if [ -f "README.md" ] && [ -f "README.adoc" ]; then
47-
echo "::error::Duplicate README: keep README.md (required Markdown); remove README.adoc."
48-
DUPLICATES=$((DUPLICATES + 1))
50+
if grep -q "GENERATED from README.adoc" README.md 2>/dev/null; then
51+
echo "✓ README.md is a derived artefact (GENERATED banner present)"
52+
else
53+
echo "::error::README.md alongside README.adoc is not a derived artefact (no GENERATED banner). Under ADR-004, README.adoc is canonical; either derive README.md via readme-derive-reusable.yml (with a [publishing.readme] declaration) or remove the hand-maintained README.md."
54+
DUPLICATES=$((DUPLICATES + 1))
55+
fi
4956
fi
5057
5158
# CONTRIBUTING can have both but .md should just be a redirect
@@ -64,15 +71,15 @@ jobs:
6471
6572
- name: Check documentation format
6673
run: |
67-
# Files that MUST be .md for GitHub community-health / profile / Glama:
68-
# README.md, SECURITY.md, CONTRIBUTING.md (can redirect), CODE_OF_CONDUCT.md, CHANGELOG.md
69-
70-
# README must be Markdown. A lone README.adoc renders as raw markup in
71-
# community-health/profile/Glama — nudge to convert (warning during the
72-
# estate-wide migration; the duplicate case above is the hard error).
73-
if [ -f "README.adoc" ] && [ ! -f "README.md" ]; then
74-
echo "::warning::README.adoc without README.md — README must be Markdown (renders raw as AsciiDoc in community-health/profile/Glama). Convert to README.md."
75-
fi
74+
# ADR-004: README.adoc is canonical estate-wide. A lone README.adoc
75+
# is correct and needs no action — do NOT nudge it to Markdown. A
76+
# README.md is only produced by deriving it (readme-derive-reusable.yml)
77+
# for repos that declare a [publishing.readme] consumer. GitHub renders
78+
# .adoc natively; only external Markdown-only consumers (Glama, the
79+
# profile repo) need the derived .md.
80+
81+
# Community-health files that GitHub special-cases by exact .md name:
82+
# SECURITY.md, CONTRIBUTING.md (can redirect), CODE_OF_CONDUCT.md, CHANGELOG.md
7683
7784
# Check other docs are .adoc
7885
for doc in ARCHITECTURE ROADMAP PHILOSOPHY INSTALL; do
Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
# SPDX-License-Identifier: MPL-2.0
2+
# README single-source derivation — Reusable Workflow (ADR-004)
3+
#
4+
# Author READMEs once in AsciiDoc; derive README.md ONLY for repos that
5+
# declare a Markdown consumer in their descriptiles. This workflow reads that
6+
# declaration, regenerates the derived file behind a fail-closed vocabulary
7+
# gate, and (in the default check mode) fails-and-tells if the committed
8+
# README.md is stale — it never self-commits and never converts an undeclared
9+
# repo. Absence of a declaration is a machine-readable "do not convert": the
10+
# anti-runaway guard that the June 2026 estate-wide .adoc->.md sweep lacked.
11+
name: README Derive Reusable Workflow
12+
13+
on:
14+
workflow_call:
15+
inputs:
16+
runs-on:
17+
description: Runner label
18+
type: string
19+
required: false
20+
default: ubuntu-latest
21+
threshold:
22+
description: Vocabulary-preservation floor (percent). ADR-004 default 98.
23+
type: number
24+
required: false
25+
default: 98
26+
standards-ref:
27+
description: Ref of hyperpolymath/standards to source the vocab-gate script from.
28+
type: string
29+
required: false
30+
default: main
31+
32+
permissions:
33+
contents: read
34+
35+
# Read-only check workflow — safe to cancel superseded runs.
36+
concurrency:
37+
group: ${{ github.workflow }}-${{ github.ref }}
38+
cancel-in-progress: true
39+
40+
jobs:
41+
derive-readme:
42+
name: Derive & verify README.md
43+
runs-on: ${{ inputs.runs-on }}
44+
timeout-minutes: 10
45+
steps:
46+
- name: Checkout caller repository
47+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4
48+
49+
- name: Read publishing declaration
50+
id: decl
51+
run: |
52+
# Prefer descriptiles; fall back to the deprecated 6a2 dir during the
53+
# estate migration. No file / no [publishing.readme] table / derive
54+
# not true => this repo has opted out and we do nothing.
55+
ECO=""
56+
for c in .machine_readable/descriptiles/ECOSYSTEM.a2ml \
57+
.machine_readable/6a2/ECOSYSTEM.a2ml; do
58+
if [ -f "$c" ]; then ECO="$c"; break; fi
59+
done
60+
61+
if [ -z "$ECO" ]; then
62+
echo "No ECOSYSTEM.a2ml found — repo not opted in. Nothing to derive."
63+
echo "derive=false" >> "$GITHUB_OUTPUT"
64+
exit 0
65+
fi
66+
echo "Reading declaration from: $ECO"
67+
68+
# Extract the flat [publishing.readme] table into KEY=VALUE lines.
69+
eval "$(
70+
awk '
71+
/^[[:space:]]*\[/ { inblk = ($0 ~ /^[[:space:]]*\[publishing\.readme\][[:space:]]*$/) ? 1 : 0; next }
72+
inblk && /=/ {
73+
key = $0; sub(/=.*/, "", key); gsub(/[[:space:]]/, "", key)
74+
val = $0; sub(/^[^=]*=[[:space:]]*/, "", val)
75+
sub(/[[:space:]]*(#.*)?$/, "", val) # strip trailing comment/space
76+
gsub(/^"|"$/, "", val) # strip surrounding quotes
77+
if (key ~ /^(canonical|derive|format|path|consumer)$/)
78+
printf "DECL_%s=%s\n", toupper(key), val
79+
}
80+
' "$ECO"
81+
)"
82+
83+
if [ "${DECL_DERIVE:-false}" != "true" ]; then
84+
echo "[publishing.readme] absent or derive != true — repo not opted in."
85+
echo "derive=false" >> "$GITHUB_OUTPUT"
86+
exit 0
87+
fi
88+
89+
CANONICAL="${DECL_CANONICAL:-README.adoc}"
90+
DERIVED_PATH="${DECL_PATH:-README.md}"
91+
FORMAT="${DECL_FORMAT:-gfm}"
92+
CONSUMER="${DECL_CONSUMER:-unspecified}"
93+
94+
if [ ! -f "$CANONICAL" ]; then
95+
echo "::error::Declared canonical '$CANONICAL' does not exist."
96+
exit 1
97+
fi
98+
if [ "$FORMAT" != "gfm" ]; then
99+
echo "::error::Unsupported derived format '$FORMAT' (only 'gfm' is implemented)."
100+
exit 1
101+
fi
102+
103+
echo "Opted in: consumer=$CONSUMER $CANONICAL -> $DERIVED_PATH ($FORMAT)"
104+
{
105+
echo "derive=true"
106+
echo "canonical=$CANONICAL"
107+
echo "derived_path=$DERIVED_PATH"
108+
echo "consumer=$CONSUMER"
109+
} >> "$GITHUB_OUTPUT"
110+
111+
- name: Install toolchain
112+
if: steps.decl.outputs.derive == 'true'
113+
run: |
114+
sudo apt-get update -qq
115+
sudo apt-get install -y -qq pandoc
116+
# asciidoctor + include-reducer from rubygems (ubuntu ships ruby)
117+
sudo gem install --no-document asciidoctor asciidoctor-reducer
118+
pandoc --version | head -1
119+
asciidoctor --version | head -1
120+
121+
- name: Fetch vocab-gate script from standards
122+
if: steps.decl.outputs.derive == 'true'
123+
run: |
124+
# Version-locked to the same standards ref that owns this workflow.
125+
git clone --depth 1 --branch "${{ inputs.standards-ref }}" \
126+
https://github.com/hyperpolymath/standards.git "$RUNNER_TEMP/standards"
127+
chmod +x "$RUNNER_TEMP/standards/.github/scripts/readme-vocab-gate.sh"
128+
129+
- name: Derive Markdown from AsciiDoc
130+
if: steps.decl.outputs.derive == 'true'
131+
env:
132+
CANONICAL: ${{ steps.decl.outputs.canonical }}
133+
run: |
134+
set -euo pipefail
135+
# 1) flatten includes so the source is self-contained
136+
asciidoctor-reducer "$CANONICAL" -o "$RUNNER_TEMP/reduced.adoc"
137+
# 2) reliable route: asciidoc -> docbook -> gfm (direct converters lose structure)
138+
asciidoctor -b docbook5 -a attribute-missing=drop \
139+
"$RUNNER_TEMP/reduced.adoc" -o "$RUNNER_TEMP/reduced.xml"
140+
pandoc -f docbook -t gfm --wrap=preserve \
141+
"$RUNNER_TEMP/reduced.xml" -o "$RUNNER_TEMP/body.md"
142+
143+
# 3) stamp: carry the canonical SPDX line (rewritten as an HTML
144+
# comment) + a GENERATED banner so no one hand-edits the artefact.
145+
SPDX="$(grep -m1 -oE 'SPDX-License-Identifier:[[:space:]]*[^[:space:]]+' "$CANONICAL" || true)"
146+
{
147+
[ -n "$SPDX" ] && echo "<!-- $SPDX -->"
148+
echo "<!-- GENERATED from $CANONICAL by standards/.github/workflows/readme-derive-reusable.yml — do not edit. -->"
149+
echo ""
150+
cat "$RUNNER_TEMP/body.md"
151+
} > "$RUNNER_TEMP/derived.md"
152+
153+
- name: Vocabulary-preservation gate
154+
if: steps.decl.outputs.derive == 'true'
155+
env:
156+
CANONICAL: ${{ steps.decl.outputs.canonical }}
157+
run: |
158+
"$RUNNER_TEMP/standards/.github/scripts/readme-vocab-gate.sh" \
159+
"$CANONICAL" "$RUNNER_TEMP/derived.md" "${{ inputs.threshold }}"
160+
161+
- name: Freshness check (fail-and-tell)
162+
if: steps.decl.outputs.derive == 'true'
163+
env:
164+
DERIVED_PATH: ${{ steps.decl.outputs.derived_path }}
165+
CANONICAL: ${{ steps.decl.outputs.canonical }}
166+
run: |
167+
set -euo pipefail
168+
if [ ! -f "$DERIVED_PATH" ]; then
169+
echo "::error::Declared derived README '$DERIVED_PATH' is missing. Regenerate and commit it."
170+
SHOW_CMD=1
171+
elif ! diff -q "$RUNNER_TEMP/derived.md" "$DERIVED_PATH" >/dev/null; then
172+
echo "::error::'$DERIVED_PATH' is STALE relative to '$CANONICAL'. Regenerate and commit it."
173+
echo "----- diff (committed vs regenerated) -----"
174+
diff "$DERIVED_PATH" "$RUNNER_TEMP/derived.md" || true
175+
SHOW_CMD=1
176+
else
177+
echo "✓ $DERIVED_PATH is up to date with $CANONICAL"
178+
exit 0
179+
fi
180+
181+
cat <<'EOF'
182+
183+
To regenerate locally (requires asciidoctor, asciidoctor-reducer, pandoc):
184+
185+
asciidoctor-reducer README.adoc -o /tmp/reduced.adoc
186+
asciidoctor -b docbook5 -a attribute-missing=drop /tmp/reduced.adoc -o /tmp/reduced.xml
187+
pandoc -f docbook -t gfm --wrap=preserve /tmp/reduced.xml -o /tmp/body.md
188+
# then prepend the SPDX + GENERATED banner (see the workflow's stamp step)
189+
190+
Fail-and-tell mode (ADR-004): CI does not auto-commit. Commit the
191+
regenerated README.md yourself.
192+
EOF
193+
exit 1

a2ml-templates/ECOSYSTEM.a2ml.template

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,25 @@ descriptions = [
6060
"{{NOT_2}}",
6161
]
6262

63+
# ── Publishing (ADR-004: README single-source derivation) ──────────────────
64+
# OPTIONAL. Omit this table entirely unless this repo has a Markdown-only
65+
# consumer (e.g. the Glama MCP directory, or the GitHub profile repo) that
66+
# cannot render the canonical README.adoc.
67+
#
68+
# Absence = "do not derive": no README.md is generated and estate sweeps must
69+
# leave README format alone. When present with derive = true, the
70+
# readme-derive-reusable.yml workflow regenerates `path` from `canonical`
71+
# behind a vocabulary-preservation gate, and fails CI if the committed file is
72+
# stale. The derived file is a build artefact — never hand-edit it.
73+
#
74+
# [publishing.readme]
75+
# canonical = "README.adoc" # authored source (canonical)
76+
# derive = true # opt in
77+
# format = "gfm" # only GitHub-Flavored Markdown implemented
78+
# path = "README.md" # derived artefact to keep fresh
79+
# consumer = "glama" # glama | github-profile | ...
80+
# reason = "Glama MCP directory renders Markdown only"
81+
6382
# ── Provenance ─────────────────────────────────────────────────────────────
6483
[provenance]
6584
# Audit trail — updated automatically by agents/bots on every write.

0 commit comments

Comments
 (0)