block-proposal-compression: price the reference prefix length instead of bounding it - #8
Open
megonen wants to merge 13 commits into
Open
block-proposal-compression: price the reference prefix length instead of bounding it#8megonen wants to merge 13 commits into
megonen wants to merge 13 commits into
Conversation
REFERENCE_PREFIX_LENGTH was argued from a worst-case bound. This measures it instead, against the real logos-blockchain code at 40e76c8 -- the real Mantle encoding, the real mantle_txhash, the real Merkle block_root and the real Block::reconstruct, all pulled in as a pinned git dependency. The threat model separates the two costs that were being conflated. Generation is offline and cheap per candidate, because mantle_txhash covers the MantleTx and not the op_proofs, so grinding needs no signature or proof; it is what sets the parameter. Propagation is noted and then set aside, deliberately, because an adversary spamming precomputed candidates from many nodes is not bounded by any single node's ingest rate, and treating one as a bound would overstate the margin. Four things are measured rather than assumed: - R_gen, single core and aggregate. The attacker's loop lands within 3% of raw Blake2b, so grinding is pure hashing with no software headroom left. Multi-core scaling is measured too -- 8 threads on an M3 give 5.39x, not 8x -- since multiplying would have understated the time an adversary needs by ~48%. - The birthday model itself, by grinding real mantle_txhash output at prefix lengths short enough to collide in seconds. Across 24 bits of doubling every measured/predicted ratio is consistent with 1.0, which is what licenses extrapolating to b = 64 and b = 128. - Reconstruction latency vs. collision multiplicity. Each combination re-hashes the whole block, so the defender's cost is O(n) per combination and 2^k overall against the attacker's sqrt(k). An M3 crosses the 1 s slot at k = 10. - Proposal sizes, which reproduce both anchors exactly: 8,555 bytes at L = 8 (the implementation's own pinned test) and 16,747 at L = 16 (the figure in #389). Conclusion: keep 16. A $10k adversary buys 64 colliding pairs at L = 14 and needs 10; at L = 16 the same budget buys 0.001 of one. That locates the crossover without a judgement call. Two findings worth flagging separately. The merged implementation is at REFERENCE_PREFIX_BYTES = 8, the one value this analysis rules out outright -- under a second of GPU time per colliding pair. And #389's "58 years / 214 days" omit the sqrt(pi/2) factor in the expected first-collision count; corrected they are 73 years / 268 days, which moves in the safe direction and changes no conclusion. RPi5 is the target validator class and its cells are left marked _(pending)_. The suite is pinned by rust-toolchain.toml and builds from source on each machine, so the Pi runs identical code with no changes.
The cost model treated k colliding pairs as buying an ongoing stall. It does not. A pair yields ambiguity only while both of its transactions sit in the mempool: once the proposer includes one, the block is applied and that transaction is removed, leaving the other unambiguous. So k pairs buy one slot, and holding a stall open across N slots needs k*N pairs. Cost still grows as sqrt of the total, but at a 1 s slot that discount does not keep up. Pricing it properly changes which conclusions the data actually supports. It makes the case against short prefixes far stronger -- a full day of stalled block production costs $0.07 at L=8, $17.79 at L=10 and $4,554 at L=12, none of which survive any reasonable reweighting of the GPU assumptions. It also makes the case against L=14 weaker, not stronger: sustaining the stall there costs $1.2M/day, which is a real deterrent and cannot be argued away with this data. So the recommendation is restated at the strength the evidence carries. L<=12 is disqualified by measurement. The choice between 14 and 16 is a judgement about margin -- assumption risk in the one unmeasured input (GPU hash rate), the lifetime of a consensus constant against future hardware, and the asymmetry between shortening L later and lengthening it after mainnet -- and the report now says that instead of implying 14 is exploitable today. 16 is recommended as 2 KB of insurance, and 14 is noted as defensible if adopted deliberately with the one-off griefing cost written down as accepted risk. The sustained-cost table is derived in analyse.py alongside the rest, so it re-prices with the GPU assumptions rather than being hand-computed.
Two gaps against the review, both closed here. Hansie's ~0.3 tx/s mempool-flood measurement was not addressed at all. The earlier draft argued propagation away in the abstract, which reads as ignoring the one piece of evidence the review actually cited. It is now engaged with directly, and the answer is structural rather than dismissive: the attack does not need volume, it needs 20 transactions. Grinding is offline and the ~10^19 candidates are discarded locally -- only the colliding pairs are ever transmitted. At 0.3 tx/s, injecting 20 transactions from one node is ~67 seconds, and sustaining 20 tx/s takes ~67 nodes. So the figure is real but composes into nothing: it is a per-node rate applied to a payload of 20, against an adversary who parallelises across nodes. The review asked how many cores or GPUs are needed; the report answered in dollars and wall-clock. Added a table in those units: to manufacture the k=10 pairs that stall one slot within a day it takes 1 GPU at L=12, 331 at L=14 and 84,619 at L=16. Core counts are included as asked, with the caveat that nobody would grind Blake2b on CPUs here. Also added a coverage table up front mapping each point of the review to the section that answers it, including the two that resolve against the earlier draft -- the concession that L<=12 is ruled out by measurement while 14-vs-16 is a judgement about margin, and the RPi5 run still outstanding.
The recommendation leaned on "shortening L later is a clean parameter change", which is wrong. L sets the References layout, so changing it in either direction is a breaking wire-format change needing a coordinated upgrade -- #389 says exactly that of the original change. Shortening is not a free knob and the argument should not have implied it was. The asymmetry that does hold is urgency and information, not cost. Learning that 16 was generous happens at leisure, from analysis, and bundles into a scheduled upgrade. Learning that 14 was thin happens while someone is stalling the chain, and the fix then ships under pressure against a mempool already seeded with colliding transactions that do not expire on their own. The corollary cuts the other way and is worth stating: since shortening costs a hard fork to save 2 KB, nobody realistically ever spends one on it, so the value shipped is in practice the value forever. That argues for getting it right now rather than relying on revisability. Also sharpened why generation, not distribution, has to set the parameter. It is not merely that generation is faster. It is that L does not affect distribution at all: a colliding pair is two transactions at any prefix length, so doubling L multiplies generation cost by 2^(8L/2) and leaves distribution untouched. Distribution is not a lever this parameter can pull, which is why no propagation measurement can stand in for a generation one.
The report leaned on Hansie's ~0.3 tx/s without ever checking whether the current code can produce a number like that. It can't, and now the report says so with a measurement instead of an argument. Two costs, both on the same pinned commit as everything else: - mempool admission (decode + preverify) is 1.44 us/tx, ~693k tx/s on one core. Signature verification is not on this path -- the mempool item is SignedMantleTx<Preverified>, whose Deserialize runs preverify(), and for a Transfer that is structural only. - the ZK multi-signature verify is 4.69 ms/tx, ~213 tx/s. That is a Groth16 verification (lb_zksign::verify) and it runs at block application, needing the UTXO set. So 0.3 tx/s is ~700x below the most expensive per-transaction crypto in the pipeline and ~2,000,000x below the admission path. Whatever produced it was not protocol CPU cost, which is the concrete reason it cannot be carried into a security argument. The experiment itself is not reproducible from the figure alone -- no harness, and it is unclear whether it measured submission, admission or inclusion -- and the report says that rather than implying otherwise. Flagged separately, outside this study's scope: at 4.69 ms/tx a full 1024-transaction block costs ~4.8 s of single-core verification against a 1 s slot, which suggests real blocks are much smaller than MAX_BLOCK_TXS. That would make the n=128 reconstruction curve the representative one (crossover k=13 rather than k=10) and raise the attacker's bar. Left as an observation, not folded into the recommendation, since it is single-core and possible parallelisation or batching was not investigated.
The section on the pending RPi5 columns gave the direction (attack gets cheaper) but not the magnitude, which is the part that decides whether the recommendation can be acted on before the Pi run. The two axes scale differently and that is the whole point. Attacker cost grows as sqrt(k) in colliding pairs, so even halving the crossover k -- from the M3's 10 to 5 -- makes the attack only ~1.41x cheaper, and the likelier drop to 8 is worth ~11%. Attacker cost grows as 2^(8L/2) in prefix length, so every 2-byte step in L is a factor of 256. One step in L outweighs any plausible change in k by more than two orders of magnitude; it would take a machine ~65,000x slower than the M3 at reconstruction to move the crossover one 2-byte step. So the Pi run is a confirmation and a completeness requirement, not a derivation. Said what it is actually for: absolute headroom in normal operation on the target validator class, the review's explicit "on a single machine RPi5" condition, and the place a surprise would surface. Also recorded a cross-check for when the data lands. The nonce sequence and hashes are deterministic, so birthday.csv must be identical between mac/ and rpi5/ except the rate column. If the draw counts differ, the harness is misbehaving on one machine and nothing else from that run should be trusted.
The crossover did not move: the Pi is ~1.7x slower row-for-row but k=9 stays under the 1 s slot at 925 ms, so the first over-slot k is 10 on both machines (13 at n=128). The stall-cost columns are therefore identical to the mac ones. Birthday determinism cross-check passed; no throttling; figure regenerated with both machines.
The RPi5 crossing the slot at the same k = 10 as the M3 reads like a clean confirmation. It is, but only by 7.5%: the Pi's k = 9 lands at 925 ms against a 1,000 ms slot. A marginally slower board, a validator under concurrent load, a throttling unit or a slightly larger block would all report k = 9, and that would be a correct measurement rather than a failed replication. Says so explicitly so nobody re-running this treats a k = 9 result as a contradiction. It changes nothing in the decision -- k = 9 makes the attack sqrt(9/10) ~= 5% cheaper, invisible against the 256x per 2-byte step in L -- which is exactly why it costs nothing to be candid about it.
The report called 10^10 H/s "roughly 2x above published hashcat Blake2b throughput for a single RTX 4090", i.e. deliberately generous to the attacker. That is wrong, and wrong in the unsafe direction: published hashcat benchmarks put a 4090 at ~1.25e10 H/s on BLAKE2b, so the tables understate one current consumer GPU by ~20% rather than overstating it by 2x. Two further things the claim got wrong. #389 does not name a device at all -- the RTX 4090 attribution was ours, added on top of #389's bare 10^10 figure, so it read as sourced when it was not. And the "2x" was never checked against a benchmark before being written down. Re-pricing at 1.25e10 scales every cost and duration by 0.8x: a day-long stall costs ~$933k at L=14 instead of $1.2M, ~$239M at L=16 instead of $298.5M. No conclusion changes at any prefix length, so 10^10 is kept for comparability with #389 rather than silently diverged from, with the divergence and its effect now stated inline and in analyse.py. Worth noting what this is an instance of. Section 8 rests part of the case for 16 on assumption risk -- that at L=14 a factor of a few in the GPU rate moves the numbers across the line that matters. This is that risk showing up in our own report, from one unchecked factor, before any future hardware is considered.
…otation Renames to the convention agreed in review: reports/block-proposal -> reports/block-proposal-compression, tools/benchmarks/reference-prefix -> tools/benchmarks/block-proposal-compression. Same name in both trees, matching how pqc is already laid out, and named for the topic rather than this one study so further compression work can sit beside it. Internal links updated; the suite still builds and its 7 tests pass from the new path. The crate keeps the name reference-prefix-bench, since that is what it measures -- the directory is the topic, the crate is the study. Said so in the suite README so the mismatch doesn't read as an oversight. Adds a Notation and terms section to the report. It used L, b, k, n, N and R_gen throughout without ever defining them, and n vs N were doing very different jobs while looking alike. Also defines the domain terms a reader meets in order -- reference, reconstruction, prefix collision, birthday collision, slot, grinding -- and states the two conventions the whole report runs on: attacker priced on strong hardware and validator on weak, and every figure labelled measured or assumed. The point is that this should still be readable months from now without reconstructing the shorthand from context.
Audit pass over the report ahead of review. Every number was re-checked against results/*.csv and against the pinned logos-blockchain commit. Numbers that were wrong: - The R_gen table carried mac figures from an earlier run. Corrected against the final run_all.sh output: node path 2.42 -> 2.40e6, attacker 6.19 -> 6.22e6, raw Blake2b 5.89 -> 6.01e6, and the four other places 6.19e6 was quoted in prose. RPi5 aggregate scaling 4.00x -> 3.98x. - The whole "1 machine (mac)" column of the collision-cost table was stale, by up to 12% (133 -> 122 days at L=12, 93 -> 86 years at L=14, 23,800 -> 22,000 years at L=16). The single-core column drifted slightly too. - "Assuming linear scaling would have understated the time an adversary needs by about 48%" had the relation backwards. 8/5.39 = 1.48, so the linear assumption credits the adversary with 48% more throughput than the machine delivers; the time is understated by 33%, not 48%. Restated in throughput terms, where 48% is the right number. One claim that did not survive checking. The report said the attacker path is "within 3% of raw Blake2b". The measured gap is 3.4% on the M3 and 5.1% on the Pi, and in both cases the attacker path is *faster* than the bare hash -- impossible, since it does the same hash plus an 8-byte copy. The bare-hash benchmark is evidently pessimized by its own black_box. Rewritten to say what the measurement supports: a difference that changes sign is at or below the noise floor, so the two are indistinguishable rather than close, which carries the same conclusion without asserting a precision the data does not have. Verified correct and unchanged: the birthday table, the reconstruction curve, the decision tables, all derived RPi5 figures (2.6 ms, ~380x, 925 ms, 7.5%, 1.65x), and every code claim against 40e76c8 -- REFERENCE_PREFIX_BYTES = 8, MAX_BLOCK_TRANSACTIONS = 1024, the two caps, the 1 s slot, and the pinned 8,555-byte proposal test. Prose: fixed two sentences that did not parse, two comma splices, a number disagreement, a dead section anchor (single hyphen where the em-dash heading generates two), and 93%/7.5% summing to 100.5%. Cut thirteen instances of the report announcing which of its own findings matter most, and two passages that restated the RPi5 result and the 380x headroom a second time in full.
Collaborator
|
Looks good, two things:
|
Review asked for reports/block-proposal for the report and tools/benchmarks/block-proposal/reference-prefix-length for the code, so the topic directory now holds one subdirectory per study rather than one flat directory per study. That also settles where this sits relative to the short-ID benchmark, which already publishes into reports/block-proposal. Every internal path follows the move, including the one that gained a level of nesting, and all relative links resolve. Adds the Makefile the review asked for, following the pqc convention in this repo: self-documenting `##` target comments behind `make help`, `where` printing the directories the tool actually reads and writes, and a read-only `check` that reports missing dependencies without installing anything. Three things the Makefile encodes that were previously only prose: - MACHINE defaults by probing the platform, so `make run` on the Pi lands in results/rpi5/ without being told, while still being overridable to label a run explicitly. - Measuring targets depend on `build`, never on `test`, so a failing hygiene check can't block a half-hour run -- but `test` is named as the gate that proves the harness still agrees with the real code path. - `make venv` handles PEP-668 systems that refuse a global pip install, and every later `make analyse` picks the venv up automatically. Without it the tables still print; only the figure is skipped. READMEs and the report's reproduction section now drive the make targets instead of repeating raw commands, so there is one place to change.
Collaborator
Author
Done 👍 edd17b9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measures
REFERENCE_PREFIX_LENGTHfor the compressed block proposal (logos-lips#389) rather than arguing it from a worst-case bound, following the review point that 16 bytes looks over-conservative and that the analysis should start from a threat model.Recommendation: keep 16 — but the case is narrower than "16 or bust". The data rules out 8, 10 and 12 outright. The choice between 14 and 16 is a judgement about margin, not a demonstration that 14 is exploitable, and the report says so.
What's here
reports/block-proposal-compression/reference-prefix-length.md: threat model → methodology → results → recommendation.tools/benchmarks/block-proposal-compression/: criterion benches plus the reconstruction-latency harness, withresults/mac/andresults/rpi5/complete.Everything runs the real logos-blockchain code at
40e76c8, pinned as a git dependency — the real Mantle encoding,mantle_txhash, the Merkleblock_root, andBlock::reconstruct. The two places the harness supplies its own code are documented inline and in the README.The report opens with a Notation and terms section defining every symbol it uses (
L,b,k,n,N,R_gen) and the domain vocabulary, so it stays readable without reconstructing the shorthand from context.Threat model
Two costs, priced separately, because only one is a real bound.
Generation is offline and needs no signature, proof or stake:
mantle_txhashcovers theMantleTxand not theop_proofs. This sets the parameter.Distribution is noted and set aside. Not because it is fast, but because
Ldoes not affect it at all — a colliding pair is two transactions at any prefix length. DoublingLmultiplies generation cost by 2^(8L/2) and leaves distribution exactly where it was. It is not a lever this parameter can pull, so it cannot be the axis the parameter is chosen on.On the ~0.3 tx/s mempool-flood result
That figure can't cap this attack, for two independent reasons.
The attack needs 20 transactions, not volume. Grinding is offline: the ~10¹⁹ candidates are discarded locally and only the colliding pairs are ever transmitted. At 0.3 tx/s, injecting 20 transactions from one node is ~67 seconds.
And the current code cannot produce a number like that. Measured on the same pinned commit: mempool admission (decode +
preverify) costs 1.44 µs/tx (~693k tx/s single-core). Signature verification isn't on that path — the mempool item isSignedMantleTx<Preverified>, whoseDeserializerunspreverify(), which for aTransferis structural only; the ZK proof is checked later by the statefulverifyat block application, at 4.69 ms/tx. So 0.3 tx/s is ~700× below the most expensive per-transaction crypto in the pipeline and ~2,000,000× below the admission path. Whatever produced it was not protocol CPU cost — it points at the harness or transport, not a limit an adversary inherits.The original experiment isn't reproducible from the figure alone (no harness, and it's unclear whether it measured submission, admission or inclusion), and the report says so.
Measured, not assumed
mantle_txhashoutput across 24 bits of doubling; every measured/predicted ratio is consistent with 1.0.birthday.csvis byte-identical between mac and rpi5 except the rate column, as the deterministic nonce sequence requires.The cost model, corrected
A colliding pair is spent after one slot: ambiguity requires both transactions in the mempool, and once the proposer includes one, the block is applied and that transaction is removed. So
kpairs buy one stalled slot, and holding a stall open costskpairs per slot. Cost grows as √(total), so bulk is discounted — but at a 1 s slot the discount doesn't keep up.Cost to sustain a stall (k = 10 pairs/slot, 10¹⁰ H/s, $0.50/GPU-hour). The hash rate is the one input that is assumed rather than measured, and it is not a generous one: published hashcat benchmarks put a single RTX 4090 at ~1.25 × 10¹⁰ H/s on BLAKE2b, so these figures understate one current consumer GPU by ~20%. Re-pricing at the measured rate scales everything by 0.8× and changes no conclusion, which is why 10¹⁰ is kept for comparability with #389 rather than quietly diverged from.
In hardware terms, to manufacture the k = 10 pairs that stall one slot within a day: 1 GPU at L=12, 331 at L=14, 84,619 at L=16.
Why 16, honestly
What the data proves: L ≤ 12 is indefensible. A full day of stalled block production costs $0.07 at L=8 — the value merged today. These survive any reasonable reweighting of the GPU assumptions.
What it leaves open: at L=14, sustaining a stall costs $1.2M/day. That's a real deterrent, and nobody can call 14 broken with this data. The case for 16 is about margin:
Lin either direction is a breaking wire-format change, so "shorten it later" is not free. But learning 16 was generous happens at leisure; learning 14 was thin happens while someone is stalling the chain. The corollary: since shortening costs a hard fork to save 2 KB, the value shipped is in practice the value forever.And the margin costs 2 KB — 16,747 vs 14,699 bytes, still a 1.98× reduction against master's 33,129.
If the team prefers 14, that's defensible and this data supports it against a sustained adversary. It should then be adopted deliberately, with the one-off griefing cost (~$4k per burned slot) and the hardware-lifetime exposure recorded as accepted risks.
Findings worth separate attention
REFERENCE_PREFIX_BYTES = 8(core/src/mantle/transactions/hash.rs) — the one value this analysis rules out outright. The spec says 16; the code says 8. Highest-priority item here.Also flagged: #389 v3 deletes
MAX_RECONSTRUCTION_COMBINATIONSandMAX_CANDIDATES_PER_REFERENCE, but they're still in the merged code. Either is defensible at L=16, but spec and implementation shouldn't disagree — and note that keeping the caps means a k=6 set drops honest proposals, which is cheaper to provoke than the k=10 stall. If they stay, they should be documented as a liveness trade, not a DoS defence.Out of scope, but surfaced by the measurements
At 14.9 ms/tx on the RPi5, verifying a full 1024-transaction block is ~15 s of single-core work against a 1 s slot (~4.8 s on the M3). That's a throughput question independent of this study, but it suggests real blocks are far smaller than
MAX_BLOCK_TXS— which makes the n=128 curve the representative one and raises the attacker's bar. Left as an observation: it's single-core, and possible parallelisation or batching wasn't investigated.Reproducing
Toolchain is pinned (
rust-toolchain.toml); both machines built from source with no code changes. GPU model, hash rate and price are single constants at the top ofanalyse.py, so the whole cost model re-derives under different assumptions.RPi5 run: 2026-08-15, Cortex-A76,
performancegovernor, ~56 °C,get_throttled=0x0before and after. Note the crossover match is narrow — the Pi's k=9 sits at 925 ms against the 1 s slot (7.5% margin), so a slower board or a loaded validator would report k=9. That's a correct measurement, not a failed replication, and it moves attacker cost by ~5%.