Conversation
Snapshot of megonen/pqc-bench @ d5636d1 (formerly pq-bench-rpi5), imported verbatim into tools/benchmarks/pqc/. It measures the migration cost of moving from the cryptography Logos uses today (X25519 + Ed25519) to PQ candidates: four measurement groups (liboqs, RustCrypto, aws-lc-rs pricing rows, and a TLS 1.3 handshake matrix across openssl-native, oqs-provider and rustls+aws-lc-rs), organised by migration phase, in one self-describing schema-2.0.0 results JSON. Imported here rather than into logos-blockchain-pocs (where it was proposed as commit 1382002) because this is a measurement tool, not a proof-of-concept implementation: it belongs next to the other benchmark harnesses in tools/benchmarks/. What this snapshot brings over the earlier pq-bench-rpi5-consolidated state: * narrowed sudo: the run escalates only for the CPU-governor write; measurement, in-run cargo builds and all results run as the invoking user (kills root-owned .work-*/target artifacts and cargo-as-root) * the Fedora fix series, verified in a Fedora 42 container and on a real Fedora 44 machine: x86 cpu-features path, silent-failure ERR traps + logged build steps, usable OpenSSL source-build path (libdir=lib, rpath, install_ssldirs, vendored reuse), family-aware package hints, and a permanent `make test-fedora` container gate * generalized naming; the Raspberry Pi 5 remains the baseline reference platform by definition (gate unchanged) * published dataset grown to 4 runs: RPi5 baseline + Apple M3 + Apple M4 Pro + Fedora 44 x86_64 (first x86 datapoint) Dashboard: https://megonen.github.io/pqc-bench/ (cherry picked from commit 59bef0d)
Baseline re-measured on the same Pi 5 by the narrowed-sudo runner: medians agree with the previous baseline within ±0.2% (primitives) and ±0.6% (TLS), so rasberrypi5-20260720T143239Z is retired to history and rasberrypi5-20260730T212145Z becomes the published reference run. Upstream commit 84946c4. (cherry picked from commit 7b0bfc1)
A benchmark's output is a report, not part of the harness. Every run now writes its JSON to reports/pqc/results and figures to reports/pqc/figures, alongside a reports/pqc/README.md that records the published dataset, its provenance and the headline migration cost. One resolver decides the location — pqb_results_dir() in setup/lib_platform.sh, mirrored in merge.py and plot.py — so run.sh, the selftest, the container gate and the analysis scripts can never disagree about where results are. PQC_RESULTS_DIR / PQC_FIGURES_DIR override it, which is what a standalone copy on a measurement box needs; the Makefile exports both so `make RESULTS=… run` redirects every target at once, and `make where` prints what will actually be used. Consequences worth noting: * per-run scratch stays in the tool (.work-* moved out of the results directory): the report tree holds finished measurements only * the publish-by-whitelist rule moved with the data, to reports/pqc/results/.gitignore, so ad-hoc local runs still cannot drift into the published set by accident * `make test-fedora` mounts the results tree into the container read-only as a second volume — the hygiene checks need the published set and the schema-1.0.0 fixture * dashboard/data/merged.json deliberately stays in the tool: it is the dashboard's input, fetched by relative path, not a result * the repo-root .gitignore has a bare `lib/` rule that would have swallowed bench/lib/; the tool's .gitignore now negates it Verified on macOS/aarch64: `make test` 21/21 green, and `make merge` reproduces the committed merged.json byte-for-byte from the new location. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 788bd72)
RUNNING-ON-YOUR-RPI5.md walked through clone/check/build/run as prose, which is exactly what `make check|build|test|run` already do — so it could only ever drift out of date, and it had: it still named the repo pq-bench-rpi5 and described the pre-narrowed-sudo run. Deleted rather than repaired. What it carried that the targets genuinely cannot — the physical measurement conditions the baseline-grade gate checks (active cooling, the 27 W PSU), the Debian 13 / OpenSSL 3.5 note that spares a source build, and the advice to build inside tmux — is folded into the README's Raspberry Pi section, which now points at the contribution gate instead of a parallel walkthrough. Also completes the rename to `pqc`: the tool is a benchmark, not a proof-of-concept implementation, and it no longer carries a name that pins it to one board. The remaining pq-bench-rpi5 strings in the config, dashboard and Cargo manifests are gone. Internal `pqb_` shell helpers and the `pqb-rust*` binary names are left alone deliberately — they are private to the harness, and renaming them would churn the build and test paths without changing anything a reader sees. Verified: `make test` 21/21 green after the rename. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit bb259ff)
The benchmark measures the cost of migrating Logos to post-quantum
cryptography. It was written on and around a Raspberry Pi 5, and the prose
and identifiers had absorbed that: "the RPi5 measurement target", "the RPi5
baseline", TARGET_CFLAGS_RPI5, a dashboard titled "RPi5 baseline",
"Contributing your RPi5 results". Read cold, it looked like a benchmark
about a single board that other machines were tolerated on.
Inverted throughout: the tool runs anywhere, and one designated *reference
platform* — currently a Raspberry Pi 5, as representative validator-grade
hardware — carries the comparable numbers. Everything else is a
cross-platform datapoint, labelled, never mixed in. Board-specific facts stay
where they are genuinely board-specific (vcgencmd sensors, /proc/device-tree,
the cortex-a76 flags, the cooling and PSU conditions the gate depends on).
The gate itself keeps its meaning and its integrity. The reference platform
moves into named constants in assemble.py rather than a config key, and the
comment now says why it is hardcoded: reference-grade must not be something a
run can claim by editing a file. Gate messages become platform-generic
("NOT reference-grade (Raspberry Pi 5): host is not the reference
platform..."); no code matches on those strings, only on the boolean, and the
schema is unchanged, so the committed results stay valid.
Carries two related changes, from asking what the run actually needs root for:
* reports/pqc/sudo-and-measurement-conditions.md — the assessment. One step
escalates (the governor write); pinning, the thermal trace and everything
else are unprivileged. What it buys is a constant clock, evidenced by the
reference run's zero frequency spread over 1659 samples. What a run without
it loses is variance control and comparability, not completeness — and the
dataset cannot supply a slowdown figure, because every committed run was
measured at `performance`, so the doc gives the A/B recipe instead of a
number it would have to invent.
* pqb_set_governor_performance now probes before it writes. A machine that
pins the governor at boot needs no privilege at all, but was still being
told it "could not set governor to performance" — the run passed the gate
and the warning was pure noise. That path is the recommendation, so it
should not look like a failure.
Verified: `make test` 21/21 green, and `make merge` still reproduces the
committed merged.json byte-for-byte.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8fd017a)
`make run` measures one operation at a time on one pinned core. That cannot answer the question a protocol designer actually has: when both peers run flat out at once, how does the work split between the one that sent the message and the one that has to consume it? If consuming costs more than producing, a peer can impose more work than it performs, and the receiving side is where a network falls over — at a rate set by the ratio, not by either side's speed. New harness, bench/stress/stress_roles.c, driven by ./stress.sh (`make stress`). Encoder produces the wire object, decoder consumes it: encaps/decaps for a KEM, sign/verify for a signature. Three phases — one thread per role, every core per role, and one encoder against a full set of decoders concurrently — plus a per-session variant that charges the ephemeral keypair to whichever side actually generates one. X25519 is in the sweep as a control, not for its numbers: both peers run the identical operation, so anything but ~1.0 is a bug in the role plumbing rather than a finding. `make test` asserts it (26 checks, all green). It earned its keep immediately — it caught the per-session metric charging keygen to the decoder alone, which is right for a KEM but wrong for a DH exchange where both peers generate a share, and which had been reporting 1.87 for an exchange that is symmetric by construction. Two more defects found while validating: * Classic McEliece SIGBUSed its worker threads. Its parameter sets keep multi-megabyte arrays on the stack; bench_pq never noticed because the main thread's stack grows on demand, but a pthread stack is a fixed mapping. Worker stacks are now 32 MB, with a regression test. * the contended phase reported the raw rate ratio of 1 encoder against T decoders, which mostly reports T — a *symmetric* algorithm scored 14 on a 14-core box. Now normalised per decoder thread, so it answers "how many decoder cores does one encoder keep busy", and it agrees with the isolated ratio across every algorithm. Stress output carries is_stress_grade and never is_baseline_grade: it uses every core and is deliberately unpinned, so it cannot meet the reference gate, and a distinct field name is what stops such a file being merged into the reference dataset by something that only checks a flag. Findings are written up in reports/pqc/sender-receiver-asymmetry.md, with the first sweep committed alongside. The headline: migrating to PQ signatures reverses the asymmetry. Ed25519 makes the receiver pay 2.35x the sender; ML-DSA 0.28-0.38, Falcon 0.15, SLH-DSA-128s 0.001. The report is explicit that these are valid-input costs and a denial-of-service attacker sends garbage, so the rejection path is the next thing to measure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 80a3008)
…ents Six defects, found by reviewing the stress harness and by exercising three paths that had been written but never run. In the harness itself: * the sink that stops the compiler eliding the crypto was ONE GLOBAL, written by every worker on every operation. A data race, and worse for a concurrency harness: a single cache line ping-ponging between fourteen cores, injecting contention into exactly the thing being measured. Now per-worker, with the workers cache-line aligned and posix_memalign'd so neighbours cannot share a line either. * latency reservoirs were pooled by concatenation, weighting a slow worker's samples equally with a fast one's. On a machine with performance and efficiency cores an E-core worker fills the same size reservoir from a third of the operations. Now pooled in proportion to operations completed. * run_phase took workers BY VALUE, so building the contended phase copied two workers that then aliased the same sample buffers with independently drifting counters. It happened to work, given what was read afterwards. Now takes pointers. Found by writing a test for the no-privilege governor path, which the sudo assessment recommends but nothing exercised: * pqb_set_governor_performance reported the machine's governor by reading cpu0 ALONE. On a host where cpu0 is 'performance' but another core is not, it answered 'performance', no demerit was recorded, and the run could be stamped reference-grade under a moving clock on most of its cores. This one is long-standing, not new. It now reports the effective governor — 'performance' only if every core agrees. PQB_CPUFREQ_ROOT lets the selftest drive both cases from a fixture, so a Linux-only path is verified anywhere. Found by exercising `make figures` and re-reading the container gate: * the Makefile's `export PQC_RESULTS_DIR := $(RESULTS)` overrode the environment, so the results mount added to `make test-fedora` would have been silently ignored inside the container. An inherited value now wins. * `make test-fedora` would have rsync'd a host-built stress_roles into the Linux container, where make would skip the rebuild and try to execute a Mach-O binary — the trap the other harness binaries are already excluded for. Closes the limitation the report flagged rather than documenting it: the rejection path is now measured. Each decoder is timed against a corrupted wire object — a valid message with a bit flipped, free for an attacker to produce and as deep a rejection as the structure allows. The answer is uniform: rejecting costs 0.91-1.08x of accepting across all 29 algorithms that have a rejection path, because these are constant-time implementations. The honest-peer asymmetry therefore transfers directly to the adversarial case, and the report no longer has to hedge about it. That also yields the figure that actually bounds a flood — receiver nanoseconds per byte the attacker sends, since an attacker spends bandwidth rather than CPU — and it points the same way as everything else: Ed25519 buys 875 ns/byte against ML-DSA-65's 23. X25519 has no rejection path at all, which is not good news: every 32-byte string is a well-formed public key, so everything is processed at full cost, 594 ns/byte. Report rewritten against a fresh sweep with the corrected harness; every derived figure in it re-checked against the results file. `make test` 28/28. Still open, and stated in the report: no reference-platform run exists for either sweep, and `make test-fedora` remains unverified here — this machine has neither podman nor docker. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit c06a284)
…change Two problems with the numbers published in 80a3008, both raised in review. FIRST: the sweeps were taken while the machine was busy — I was running `make figures`, `make test` and other work alongside them. For a harness whose measurement IS throughput under saturation, that is not background noise: load arriving during one role's leg and not the other's moves that algorithm's ratio, not merely its rate. Those results are void and are replaced. Re-measured as three runs on a quiet machine (1-min load 1.66 / 1.99 / 1.64), reported as medians with the per-algorithm spread — most under 1%. A fourth sweep ran accidentally at load 113 and is kept deliberately, because it turns the concern into a measurement: absolute latency inflated 26-33% (X25519 derive 24.0 us against 18-19 us quiet) while every ratio stayed within 4%. Ratios are largely self-normalising, since load steals cycles from both roles alike; absolute figures are not. Deleting that run would leave the claim unsupported, so it is whitelisted and labelled as what it is. So the earlier per-message ratios were about right, but for no good reason — asserting them from one contaminated run was not justified. What was actually wrong is everything absolute: the per-byte amplification, the receiver-cores table, every microsecond. All were ~25-30% overstated. Runs now record `loadavg_before`/`loadavg_after`, and a run started above load 1.0 writes "machine was not idle at start" into its own not_reference_because list. Contaminating a run is easy; it should not be invisible afterwards. SECOND: "receiver CPU per attacker byte" was reported as one number per algorithm, and an exchange has more than one message. The encapsulator receives a public key; the decapsulator receives a ciphertext. For ML-KEM those are close (7.6 vs 9.2 ns/byte). For Classic McEliece they differ by a factor of 2.8 MILLION — 0.05 ns/byte flooding the encapsulator, which must be sent a 261 KB public key to buy 12 us of work, against 140 000 ns/byte flooding the decapsulator, whose ciphertext is 96 bytes. One number for that algorithm was misleading. cost_per_received_byte now prices every message, honest and attacked, with a signer explicitly recorded as receiving nothing (signing is initiated by the signer, not by a peer). The honest cost sits beside the attacked one rather than being displaced by it — asked for, and right, since it is the capacity-planning figure while the attacked one is the flood bound. analyze/asymmetry.py takes several result files and reports medians with spread. `make test` grew two schema assertions: a signer must report receiving nothing, a KEM encapsulator must report receiving a public key, and X25519 — the same operation in both directions — must price identically both ways. 28/28 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 5b7faed)
The sudo assessment promises that a machine which pins the governor at boot "does not prompt" — pqb_set_governor_performance keeps that promise at the write, but make run's front door (bench-run.sh) ran `sudo -v` before run.sh could find out, so the boot-pinned box still got a password prompt for a privilege the run would never use. bench-run.sh now makes the same every-core check first (_pqb_effective_governor, honoring the PQB_CPUFREQ_ROOT fixture hook) and skips straight to an unprivileged run when all cores already agree on 'performance'. The prompt appears only when escalating would actually change something. Both branches exercised against fixture cpufreq trees; make test 28/28 unchanged.
Collaborator
|
Pushed a small commit on top (dc91962): the sudo prompt now skips itself when the governor is already 'performance' on every core — the boot-pinned-box behavior the sudo report already promised, but bench-run.sh ran Review status from our side: code review done (the stress harness, the governor fix and the results relocation all look right — the X25519 symmetric control reading 1.00 live is a nice touch), and the full gate verified on an Apple M3: |
Closes the report's stated gap ("no reference-platform run yet") with
measured data on both sweeps, plus a third platform for the stress
ratios:
* rasberrypi5-20260811T211615Z — full measurement run of THIS tree on
the Raspberry Pi 5: reference-grade, zero warnings, 28-check gate
green, medians within ±0.4% of the published 20260730 baseline (the
third consecutive reproduction of those numbers).
* stress-rasberrypi5-20260812T043436Z — the asymmetry sweep under
reference conditions: governor performance, 1-min load 0.00 at start,
X25519 control 1.0004. Structure-driven ratios reproduce (Ed25519
2.48, ML-KEM 1.16-1.17, ML-DSA 0.28-0.37, Falcon 0.17-0.18, SLH-DSA
0.001-0.06). One nuance: Classic McEliece's decaps/encaps lands 2-3x
lower than on the M4 (307-523x vs 690-1490x) — for the table-heavy
decoder the exact ratio is platform-dependent; the disqualifying
magnitude class is what transfers.
* stress-mehmetmac-20260812T{111720,112638,113607}Z — Apple M3
median-of-three, run on an interactive machine (loads 3.78/8.05/5.86)
and still within a few percent of the quiet-M4 medians: the report's
load-robustness claim exercised on a third platform. The ML-KEM rows'
wider spread there is the documented macOS ~1 us clock quantisation,
not noise — the mean column resolves it.
The report's reference-platform paragraph is updated from prediction to
result. The 20260811 measurement run is committed as verification data;
swapping it into published_runs.txt (and rebuilding merged.json) is
left as a follow-up decision.
Collaborator
|
Reference-platform data is in (32b5e60) — the gap the report's Status section flagged is closed with measurements, not estimates:
There are just two small things left, and I'll leave them up to you @madxor. One is whether we should replace published_runs.txt with the new Pi run (which would regenerate merged.json). The other, if you're up for it, is adding one x86 stress benchmark from your Fedora machine. That's the only missing entry in the platform table. |
Rebuilds the gh-pages branch (root artifact index + /pqc/ dashboard with the published merged.json) and force-pushes it. The gh-pages branch is generated output, never hand-edited. Serving URL: https://logos-blockchain.github.io/research/pqc/ — an admin has to enable Pages once (deploy from gh-pages, /).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the post-quantum cryptography benchmark as
tools/benchmarks/pqc, with itsmeasurement record in
reports/pqc.Imported from logos-blockchain-pocs#113
with @megonen's authorship on the two snapshot commits intact. It lands here
rather than in the PoC repo because it is a measurement tool, not a
proof-of-concept implementation — it belongs beside the other benchmark
harnesses.
What it measures
The migration cost from the cryptography Logos uses today (X25519 + Ed25519) to
PQ candidates, in four groups — liboqs primitives, RustCrypto primitives,
aws-lc-rs pricing rows, and a TLS 1.3 handshake matrix across three stacks —
organised by migration phase, in one self-describing results JSON.
On the reference Raspberry Pi 5, hybrid key exchange (phase 0) costs ×1.26
handshake latency and ×2.49 handshake bytes; rustls + aws-lc-rs completes the
same handshake 2.1–3.8× faster than OpenSSL-native.
Integration changes on top of the import
reports/pqc/resultsand figures toreports/pqc/figures. One resolverdecides the location, mirrored in the shell, the Makefile and the analysis
scripts, so they cannot disagree;
make whereprints it. Verified:make mergereproduces the committedmerged.jsonbyte-for-byte from the newlocation.
make check|build|test|runas prose and had already drifted. What it uniquely carried — the physical
conditions the grade gate checks — moved into the README.
benchmark about one board that other machines were tolerated on. Inverted
throughout; the reference platform is now a named constant in
assemble.py,hardcoded on purpose so a run cannot be promoted to reference-grade by editing
a config file.
Sender/receiver asymmetry (
make stress)A second harness, answering a question the per-operation benchmark cannot: when
both sides of an exchange run flat out at once, who pays? Encoder produces
the wire object, decoder consumes it. Three phases — one thread per role, every
core per role, and one encoder against a full set of decoders.
Findings, median of three runs, full write-up in
reports/pqc/sender-receiver-asymmetry.md:the sender; ML-DSA 0.29–0.38, Falcon 0.15, SLH-DSA-128s 0.001. For nodes that
verify far more than they sign, the migration bill lands on the signer.
algorithms with a rejection path, because these are constant-time
implementations. The honest-peer ratios are also the adversarial ones.
64-byte Ed25519 signature buys 641 ns of verification; a 3309-byte ML-DSA-65
signature buys 17.5 ns/byte — 37× less.
flooding the decapsulator against 0.05 ns/byte flooding the encapsulator, a
factor of 2.8 million.
X25519 is in the sweep as a control, not for its numbers: both peers run the
identical operation, so anything but 1.0 is a bug in the role plumbing.
make testasserts it, and it earned its keep — it caught a metric that chargedkeygen to one side of a symmetric exchange.
Measurement hygiene
Stress output carries
is_stress_gradeand neveris_baseline_grade. It usesevery core and is deliberately unpinned, so it cannot meet the reference gate; a
distinct field name stops such a file being merged into the reference dataset by
anything that only checks a flag.
Runs record the load average, and one started above load 1.0 says so in its own
not_reference_becauselist. A sweep that ran at load 113 is kept deliberately:it shows competing load inflates absolute latency 26–33% while leaving the
ratios within 4%, which is the evidence for quoting ratios and not absolutes.
Verification
make testis 28/28 on macOS/aarch64, covering harness correctness, linktargets, the three TLS stacks, the stress harness, and the no-privilege governor
path against a fixture.
Two things are not verified here and are stated in the reports:
Apple M4 Pro numbers. Ratios should be structure-dominated; the per-byte
figures will be larger on a Pi.
make test-fedorais unchanged in intent but unrun — no container engine onthe machine used. Its logic was reviewed and every build artifact is excluded
from the container copy.
Also included:
reports/pqc/sudo-and-measurement-conditions.md,assessing whether the benchmark needs root. Short answer: no. One step escalates
(the CPU-governor write), and it can be removed entirely by setting the governor
at boot. Writing the test for that path exposed a long-standing bug where the
governor was read from
cpu0alone, so a partly configured machine could bestamped reference-grade under a moving clock on its other cores.
🤖 Generated with Claude Code