Integrate exact prime-field NTT CPU scan

[philsippl]

The opt-in CPU linear scan now runs the exact NTT implementation through the production service (IRIS_MPC_CPU_NTT=1). It scans every record, all dimensions, all 31 rotations, and both orientations. The final tuning reaches 13.42–13.43M comp/s through the warmed service, up 20% from a fresh 11.16M reference for the previous native-prime implementation. With the same synchronization fix, the original CPU path reaches 6.30M comp/s, making the fair end-to-end improvement 2.13×.

Stacked on #2348 (codex/cpu-linear-scan-fused-mirror). Depends on worldcoin/ampc-common#150, pinned at a1ed7358f0e77e8549b2e809b3eb5620ece30e10. The original CPU path remains the default.

The dense scan stays in F_52201: aggregate code/mask spectra into g=2*C-m, perform one selected inverse, and threshold directly in that field. Choosing public offset 32,768 makes the wrap comparisons share their low 15 bits, reducing the circuit from 46 to 31 secure ANDs in 17 layers. Only public anonymous-statistics candidates and forced targets recover original ring scores; strict matching and retained distances use the existing protocols. Pairwise PRF masks require no edaBit inventory.

Additional changes batch binary masks, serialize refreshed scores and AND outputs without cloning, and open packed predicate shares without expanding and repacking secret bits. The shared dependency reuses HTTP connections and replaces one-second waits for ordinary batch-ID/hash lag with bounded short backoff; synchronization checks and timeouts remain unchanged. This removed observed clusters of one-second stalls.

The implementation includes private conversion of deployed shares, query preprocessing, pinned workers, versioned/restartable derived PostgreSQL persistence, inserts/updates/deletions, and bounded raw caches. Spectral payload is 38,400 bytes per record for the resident eye and party. This tuning preserves cache format 2 and the score kernel. Original shares remain authoritative; disabling the flag restores the original path.

CPU measurements

Three existing AWS r8g.24xlarge hosts, Rust 1.95 with native Graviton4 flags, approximately 1.05 million records, all 31 rotations in both orientations. The service uses batch size 1, request parallelism 48 unless stated, connection parallelism 16, 4,096-record scan chunks, 256-record worker tasks, LUC 500, LFU 12,288, TLS, PostgreSQL, MPC, persistence, and all-party result delivery. Moto supplies AWS messaging/storage APIs.

Each run queued 128 fresh requests, excluded the first 16 from the steady completion interval, and validated 384 agreeing, correct party responses. Exact comparison counts include insertions. Timed runs did not overlap compilation, profiling, or kernel benchmarks.

Path	Dot/runtime cores	Matching comp/s	Queued full-service comp/s	Entire burst from publish comp/s
Previous 46-gate NTT, fresh reference	64/32	14,139,790	11,161,527	10,855,690
Final NTT	64/32	15,152,958	13,422,213	12,312,677
Final NTT, repeat	64/32	15,162,452	13,434,131	10,757,013
Final NTT, more score cores	72/24	14,535,681	12,937,411	12,465,005
Final NTT, request parallelism 96	64/32	14,613,560	12,906,976	12,315,320
Original CPU, same synchronization fix	85/11	6,397,674	6,303,490	5,991,272

Retain SMPC__SEPARATE_TOKIO_CORES_PER_NODE=32 and request parallelism 48 alongside IRIS_MPC_CPU_NTT=1. Enabling NTT alone retains the old core allocation. More score cores or more concurrent chunks were slower. One comparison is one database record, one eye, one orientation, including every dimension and all 31 rotations; rates are not multiplied by parties or rotations.

The final 64/32 completion intervals were 17.534 and 17.521 seconds versus 21.073 for the fresh reference. Throughput after warm-up agrees within 0.1%; whole-burst rates remain sensitive to initial queue wait (first response 1.98 versus 5.15 seconds). Including client preparation gives 11.30M and 9.97M comp/s. These are finite synthetic service bursts, not sustained production capacity measurements. Earlier intermediate runs with synchronization stalls are retained in the documentation.

Where the local CPU gain goes

The quiet local NTT kernel reaches 21,152,379 comp/s with 64 cores (99.145 ms), while the earlier 24,344,294 comp/s used 85 cores (86.146 ms). Both used 1,048,576 distinct resident payloads and nine measured passes after one warm-up. The integrated path additionally performs MPC, threshold opening, scheduling, candidate handling, query conversion, ingestion, and persistence. Its measured service gain is 2.13×, not 3×.

The profile identified bit transposition, PRGs, copies, allocation, and TLS outside scoring. The new circuit reduces measured traffic from approximately 563 MB to 440 MB sent per party per request (22%). On party 0, median batch reception falls from 23 ms to 1 ms; final median compute is 154 ms and persistence 2 ms. Query conversion/preprocessing measured 9.7 ms median in a separate diagnostic run. CPU sample fractions are not wall-time fractions because scoring and MPC overlap.

Validation

• Native ARM release tests with aes_rng_prf: all four NTT tests pass, including every field representative with randomized three-party shares, tiny/uneven chunks, packed tails, original-ring parity for rotations/mirroring, and inverse SIMD boundaries. Retained-score/forced-target parity also passes.
• The exact pinned dependency passes local NTT tests and cargo clippy -p iris-mpc-cpu --lib --tests --bench linear_scan_ntt_cpu --locked --no-default-features -- -D warnings.
• The dependency passes all 13 library tests and clippy. Its new HTTP test covers stale IDs, HTTP 409, stale hashes, prompt retry, and connection reuse.
• The final native binary passed 40 mixed requests / 120 agreeing responses, with no expected-result failures: uniqueness/mirror, reauthentication, reset, recovery, and deletion. Duplicate and mirror cases returned the expected match and mirror-attack flags. This used a separate small correctness database.
• Earlier integration checks passed 100 mixed requests/300 agreeing responses, four restart replays with identical matched IDs, and the native ARM PostgreSQL migration test covering cache reuse, partial-generation repair, format invalidation, and authoritative-version mismatch rejection.
• Formatting and diff checks pass. Raw logs, results, CSVs, and historical machine logs are excluded from the PR.

See implementation, complete measurements, and reproduction.