Per-deploy replay cost is still 2-4× slower on 8/16-CPU pods than on 2-CPU pods, after the HotStore O(store-size) fix

[9Neechan]

What did you expect to see?

After fixing the dominant bottleneck behind #43 (create_soft_checkpoint()/HotStore::snapshot() doing an O(store-size) clone on every deploy, fixed via a 256-way sharded HotStore — branch issue-43, commit 77b62a7a), per-deploy replay cost across CPU profiles should be flat or improve with more CPU, not still show an inversion.

What did you see instead?

Ran the parallelism suite (full MAX_DEPLOYS sweep: 1/5/10/50/100/250) on all three CPU profiles against the fixed image:

MAX_DEPLOYS	minimum (2 CPU) s/depl	mid (8 CPU) s/depl	high (16 CPU) s/depl	mid/min	high/min
1	0.168	0.387	0.234	2.30×	1.39×
5	0.108	0.354	0.293	3.28×	2.71×
10	0.097	0.383	0.359	3.95×	3.70×
50	0.129	0.290	0.373	2.25×	2.89×
100	0.139	0.377	0.351	2.71×	2.53×
250	0.071	0.186	0.193	2.62×	2.72×

minimum is faster than both mid and high at every single N in the sweep, by a consistent 1.4-4× margin. This is not noise from a single sample — the pattern holds across all six independent block-size measurements.

The absolute magnitude improved massively versus the pre-fix baseline (mid @ N=1 went from 2.1-3.7s to 0.387s, ~6-10× faster) and the old super-linear (n^1.76) runaway growth with block size is gone on every profile — so the HotStore fix is real and worth shipping. But the literal "scales backwards with CPU" claim from the original bug report still reproduces, just at much smaller absolute cost.

Steps to reproduce

1. Deploy 9neechan/rust-node:issue-43-sharded-hotstore (any build containing commit 77b62a7a).
2. Run the parallelism suite separately on minimum, mid, and high profiles (make job-run ... SUITE=parallelism PROFILE=<profile>; parallelism doesn't support ALL_PROFILES=1 yet).
3. Compare s/depl (PHASE ms / N) at any given N across the three profiles.

Supporting data

• Per-deploy Rholang execution scales backwards with CPU #43 — "Profile comparison result" section has the full before/after context; this issue is the residual split out of that investigation once the dominant bottleneck was confirmed fixed.

Why this looks like a new (separate) issue

• The dominant cost identified and fixed in Per-deploy Rholang execution scales backwards with CPU #43 (create_soft_checkpoint O(store-size)) is confirmed gone: EVAL ms collapsed 7.5× and stays flat regardless of block size or CPU profile (see Per-deploy Rholang execution scales backwards with CPU #43 sub-phase breakdown data). Whatever is causing the residual 2-4× CPU inversion is a different mechanism.
• Not explained by PRECHARGE/REFUND (the current dominant per-deploy sub-phase, per Per-deploy Rholang execution scales backwards with CPU #43's sub-phase breakdown) — those are fixed-cost PoS/RevVault contract executions unrelated to CPU count or store size.
• The magnitude (2-4×) and shape (present at every block size, not growing with N) don't match either of Per-deploy Rholang execution scales backwards with CPU #43's two already-investigated mechanisms (store-size-proportional checkpoint cost, or per-deploy contract execution cost) — it looks structural to how the runtime uses the available CPU count, not to workload size.

Root cause hypothesis

Not root-caused. Leading candidate, carried over from #43's earlier (and at-the-time inconclusive) investigation: Tokio's multi-thread runtime (Builder::new_multi_thread().enable_all()) defaults worker_threads() to available_parallelism(), which reads the pod's CPU limit (cgroup quota: 2/8/16 for minimum/mid/high) rather than the actual amount of concurrent work available (1-250 lightweight deploys per block in this benchmark). More worker threads contending over the same small set of locks/channels/RSpace structures on a trivially small workload plausibly costs more in scheduling and cache-line contention than it gains in parallelism.

This hypothesis was tested once before (TOKIO_WORKER_THREADS pinned to cgroup quota, noted in #43's "What worked" / Tokio runtime section) and called only "marginal" — but that test ran before the HotStore fix, when create_soft_checkpoint()'s O(store-size) cost dominated everything else and could have masked a smaller, secondary thread-count effect. Worth retesting now that the dominant cost is removed.

Other libraries the node depends on may also read num_cpus::get() directly regardless of TOKIO_WORKER_THREADS (e.g. casper/src/rust/engine/lfs_block_requester.rs:762, per #43), which could contribute additional thread oversubscription beyond Tokio's own pool.

Suggested fix / next steps

• Rerun mid and high parallelism sweeps with TOKIO_WORKER_THREADS pinned to 2 (matching minimum's CPU count), against the sharded-HotStore image, and compare s/depl across the sweep. If the inversion collapses, this is a second, independent, config-only fix (no code change needed) — document it as a required deployment setting or make the node itself default worker_threads() from the cgroup quota rather than host affinity.
• If pinning worker threads doesn't fully close the gap, profile (flamegraph) a mid validator during a small-N replay to see where the extra time goes with more threads active — lock contention, cache thrashing, or context-switch overhead are the likely candidates.
• Audit other direct num_cpus::get() call sites in the node for the same host-affinity-vs-cgroup-quota mismatch (e.g. lfs_block_requester.rs:762).

Related issues

• Per-deploy Rholang execution scales backwards with CPU #43 — parent investigation; this is the residual split out once the dominant create_soft_checkpoint bottleneck was confirmed fixed and shipped.