feat: [U4-B] benchmark kernels + lowering/correctness/limitations wri…

.github/workflows/ci.yml

@@ -16,6 +16,10 @@ jobs:
         run: |
           # LLVM 22 is not in the ubuntu-24.04 base repos; pull it from
           # apt.llvm.org via the official installer script.
+          # llvm-22-tools supplies llc and clang-22 the driver compiler, both
+          # needed by the benchmark checksum test (test/bench_checksum.test),
+          # which is the only part of the suite that compiles natively rather
+          # than running under lli.
           wget https://apt.llvm.org/llvm.sh
           chmod +x llvm.sh
           sudo ./llvm.sh 22

.gitignore

@@ -1,5 +1,6 @@
 # Build output
 /build/
+/bench/build/
 *.so
 *.o
 

README.md

@@ -11,6 +11,48 @@ processing — are expressible in LLVM IR (`<K x i4>`, `<K x i2>`, `<K x i1>`) b
 illegal on real targets, so the default legalizer scalarizes them and discards
 the parallelism. nybbler rewrites them into byte-vector carrier ops instead.
 
+On the benchmark kernels this is worth **13x** on an `i4` arithmetic chain and
+**58x** on an `i2` one, measured against the same backend at the same
+optimization level — see [`docs/benchmarks.md`](docs/benchmarks.md).
+
+## What it lowers
+
+| Category | Operations | Widths |
+|---|---|---|
+| Bitwise | `and`, `or`, `xor` (and `not`, which LLVM spells as `xor -1`) | i1, i2, i4 |
+| Arithmetic | `add`, `sub` | i1, i2, i4 |
+| Shifts | `shl`, `lshr`, `ashr` | i1, i2, i4 |
+| Comparisons | `icmp eq`, `ne`, `ult`, `slt` | i1, i2, i4 |
+
+Every operation goes through one shared **carrier dispatch**: pad to a byte
+multiple with zero lanes if needed, bitcast the operands to a `<M x i8>`
+carrier, run a per-operation handler, then bitcast and narrow back. Only the
+handler differs between operations, so adding one means writing a single
+function.
+
+Vectors whose bit width is not a multiple of 8 are **padded**, not skipped.
+
+Anything without a handler — `mul`, `udiv`, the remaining `icmp` predicates —
+is left untouched for the default legalizer. The full boundary is in
+[Limitations](docs/benchmarks.md#limitations).
+
+## Documentation
+
+- [`docs/lowering.md`](docs/lowering.md) — the per-operation lowerings: carrier
+  dispatch, the SWAR add/sub carry and borrow containment, per-field shift
+  masking, `ashr` sign handling, the compare lowerings.
+- [`docs/correctness.md`](docs/correctness.md) — why the bitwise case is
+  correct by construction, how the masked paths keep carries and shifted-in
+  bits inside their field, and how the correctness matrix verifies it.
+- [`docs/benchmarks.md`](docs/benchmarks.md) — benchmark methodology, measured
+  results, and limitations.
+
+## Requirements
+
+- LLVM 22 (this project pins `/usr/lib/llvm-22`; tools `opt-22`, `llc-22`,
+  `clang-22`, `lli-22`, `FileCheck-22`). On WSL2 / Ubuntu 24.04 these come from
+  the `llvm-22` packages via [apt.llvm.org](https://apt.llvm.org)
+  (`wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && sudo ./llvm.sh 22`).
 ## What it lowers
 
 | Class | Operations | `i1` | `i2` | `i4` |
@@ -117,7 +159,14 @@ This produces the plugin `build/libNybbler.so`.
 ## Run on a single file
 
 ```bash
-opt -load-pass-plugin ./build/libNybbler.so -passes=nybbler in.ll -S
+opt-22 -load-pass-plugin ./build/libNybbler.so -passes=nybbler in.ll -S
+```
+
+Example:
+
+```bash
+opt-22 -load-pass-plugin ./build/libNybbler.so -passes=nybbler \
+    test/shape/add_i4.ll -S
 ```
 
 ## Test
@@ -126,6 +175,32 @@ opt -load-pass-plugin ./build/libNybbler.so -passes=nybbler in.ll -S
 lit -v build/test/
 ```
 
+(Equivalently `llvm-lit-22 build/test/` where that wrapper is installed.) The
+suite has four layers:
+
+- `test/shape/` — 36 FileCheck tests asserting the exact carrier sequence per
+  operation per width, plus `CHECK-NOT: extractelement` to prove it did not
+  scalarize.
+- `test/diff/`, `test/pad_diff.ll`, `test/shift_overwidth.ll` — differential
+  tests running each kernel both unlowered (scalarized, the ground-truth
+  reference) and lowered under `lli`, requiring bit-identical output over
+  structured and seeded-random inputs. Reproduce a failure exactly with
+  `NYBBLER_DIFF_SEED=<n>`.
+- `test/edge_values.ll` — golden hex bytes for boundary inputs.
+- `test/coverage.test` — fails if the operation × width matrix has a hole.
+
+## Benchmark
+
+```bash
+bash bench/run.sh
+```
+
+Builds every kernel in `bench/kernels/` twice — once through `llc` alone
+(scalarized baseline) and once through `opt -passes=nybbler` followed by the
+identical `llc` line — verifies the two produce byte-identical output, then
+prints a timing table. `bash bench/run.sh --check-only` runs just the
+correctness gate; that is what CI does, since shared runners are too noisy to
+assert a speedup threshold.
 Four layers, all wired into the same `lit` invocation so CI fails if any of them
 regresses:
 

bench/driver.c

@@ -0,0 +1,142 @@
+/* driver.c -- timing and checksum harness for the nybbler benchmark kernels.
+ *
+ * Kernel-agnostic: every kernel in bench/kernels/ exports the same symbol with
+ * the same signature, so this file is compiled once per kernel and linked
+ * against either the baseline object (llc alone, narrow op scalarized) or the
+ * lowered object (opt -passes=nybbler, then the identical llc). The two
+ * binaries differ only by that pass, which is what makes the comparison fair.
+ *
+ * Build-time knob (run.sh reads it from the kernel's NYB_OUT_BYTES_PER_VEC
+ * comment): kernels that write a packed compare mask produce fewer output
+ * bytes than they consume.
+ *
+ * Usage:
+ *   ./bench_x               time it and print a result line
+ *   ./bench_x --check-only  print only CHECKSUM <hex>, no timing
+ */
+
+#define _POSIX_C_SOURCE 200809L
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <time.h>
+
+/* Input vectors are always 16 bytes: the carrier nybbler bitcasts to is
+ * <16 x i8> for every width (128 bits = 32 x i4 = 64 x i2 = 128 x i1). */
+#define VEC_BYTES 16
+
+#ifndef NYB_OUT_BYTES_PER_VEC
+#define NYB_OUT_BYTES_PER_VEC 16
+#endif
+
+#ifndef NYB_NAME
+#define NYB_NAME "kernel"
+#endif
+
+/* 256 KiB per buffer. Sized to stay resident in the 512 KiB per-core L2 of the
+ * development machine so the loop is compute-bound: at DRAM bandwidth both
+ * builds would converge on the memory system and the lowering difference would
+ * disappear into the noise. */
+#define BUF_BYTES (256u * 1024u)
+/* Exact: 262144 / 16. clang-tidy flags this as integer division reaching a
+ * floating-point context below; it divides evenly by construction. */
+#define NVEC      (BUF_BYTES / VEC_BYTES)
+
+#define WARMUP_REPS 20
+#define TIMED_REPS  50
+
+void nyb_kernel(const uint8_t *a, const uint8_t *b, uint8_t *o, uint64_t nvec);
+
+/* Deterministic fill. A fixed-seed xorshift64 rather than rand() so both
+ * builds see byte-identical inputs and the checksum is reproducible across
+ * machines and libc versions. */
+static void fill(uint8_t *p, size_t n, uint64_t seed) {
+    uint64_t s = seed;
+    for (size_t i = 0; i < n; i++) {
+        s ^= s << 13;
+        s ^= s >> 7;
+        s ^= s << 17;
+        p[i] = (uint8_t)(s >> 24);
+    }
+}
+
+/* FNV-1a over the output buffer. Two jobs: it is the correctness gate that
+ * run.sh compares between the two builds, and it makes the output live so the
+ * optimizer cannot delete the kernel loop as dead. */
+static uint64_t checksum(const uint8_t *p, size_t n) {
+    uint64_t h = 1469598103934665603ULL;
+    for (size_t i = 0; i < n; i++) {
+        h ^= p[i];
+        h *= 1099511628211ULL;
+    }
+    return h;
+}
+
+static double now_ns(void) {
+    struct timespec ts;
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return (double)ts.tv_sec * 1e9 + (double)ts.tv_nsec;
+}
+
+static int cmp_double(const void *x, const void *y) {
+    double a = *(const double *)x, b = *(const double *)y;
+    return (a > b) - (a < b);
+}
+
+int main(int argc, char **argv) {
+    int check_only = (argc > 1 && strcmp(argv[1], "--check-only") == 0);
+
+    size_t out_bytes = (size_t)NVEC * NYB_OUT_BYTES_PER_VEC;
+
+    uint8_t *a = aligned_alloc(64, BUF_BYTES);
+    uint8_t *b = aligned_alloc(64, BUF_BYTES);
+    uint8_t *o = aligned_alloc(64, out_bytes);
+    if (!a || !b || !o) {
+        fprintf(stderr, "allocation failed\n");
+        return 1;
+    }
+
+    fill(a, BUF_BYTES, 0x9E3779B97F4A7C15ULL);
+    fill(b, BUF_BYTES, 0xD1B54A32D192ED03ULL);
+    memset(o, 0, out_bytes);
+
+    for (int r = 0; r < WARMUP_REPS; r++)
+        nyb_kernel(a, b, o, NVEC);
+
+    if (check_only) {
+        printf("CHECKSUM %016llx\n", (unsigned long long)checksum(o, out_bytes));
+        free(a); free(b); free(o);
+        return 0;
+    }
+
+    /* Time each repetition separately and report the minimum. The minimum is
+     * the least contaminated sample: scheduling and interrupts can only ever
+     * add time, never remove it. The median is printed alongside so a run
+     * where the machine was busy throughout is visible rather than hidden. */
+    double samples[TIMED_REPS];
+    for (int r = 0; r < TIMED_REPS; r++) {
+        double t0 = now_ns();
+        nyb_kernel(a, b, o, NVEC);
+        samples[r] = now_ns() - t0;
+    }
+
+    uint64_t sum = checksum(o, out_bytes);
+
+    qsort(samples, TIMED_REPS, sizeof(double), cmp_double);
+    double best = samples[0];
+    double median = samples[TIMED_REPS / 2];
+
+    double ns_per_vec = best / (double)NVEC;
+    double gib_per_s = ((double)BUF_BYTES / (best * 1e-9)) / (1024.0 * 1024.0 * 1024.0);
+
+    /* One machine-parsable line; run.sh splits on whitespace. */
+    printf("RESULT %s ns_per_vec=%.4f median_ns_per_vec=%.4f gib_per_s=%.2f "
+           "checksum=%016llx nvec=%u\n",
+           NYB_NAME, ns_per_vec, median / (double)NVEC, gib_per_s,
+           (unsigned long long)sum, (unsigned)NVEC);
+
+    free(a); free(b); free(o);
+    return 0;
+}

bench/kernels/arith_i4.ll

@@ -0,0 +1,50 @@
+; arith_i4 -- nibble-packed i4 arithmetic over a byte stream.
+;
+; NYB_DESC: <32 x i4> add / shl-by-field / sub chain
+; NYB_OUT_BYTES_PER_VEC: 16
+;
+; Exercises the masked SWAR paths that nybbler exists for: lowerAdd's carry
+; containment, lowerShift's per-field boundary masking, and lowerSub's borrow
+; absorber -- three different mask formulas over the same carrier.
+;
+; Kernel shape (see docs/benchmarks.md "Why the loads are <16 x i8>"): memory
+; traffic stays at byte type and the narrow type appears only between the
+; bitcast pair around the op chain. A <32 x i4> load/store would be legalized
+; by scalarizing in *both* builds and would swamp the measurement.
+;
+; The shift amount is masked to [0, 3] in IR so every field's shift is in
+; range. An amount >= N is poison, which has no defined value for the
+; scalarized baseline to produce -- the two builds' checksums could then
+; legitimately disagree and the run would fail for a non-reason.
+
+define void @nyb_kernel(ptr noalias %a, ptr noalias %b, ptr noalias %o, i64 %nvec) {
+entry:
+  %nonempty = icmp ugt i64 %nvec, 0
+  br i1 %nonempty, label %loop, label %done
+
+loop:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+  %pa = getelementptr inbounds <16 x i8>, ptr %a, i64 %i
+  %pb = getelementptr inbounds <16 x i8>, ptr %b, i64 %i
+  %po = getelementptr inbounds <16 x i8>, ptr %o, i64 %i
+
+  %la = load <16 x i8>, ptr %pa, align 16
+  %lb = load <16 x i8>, ptr %pb, align 16
+  %va = bitcast <16 x i8> %la to <32 x i4>
+  %vb = bitcast <16 x i8> %lb to <32 x i4>
+
+  %sum = add <32 x i4> %va, %vb
+  %amt = and <32 x i4> %vb, splat (i4 3)
+  %shifted = shl <32 x i4> %sum, %amt
+  %res = sub <32 x i4> %shifted, %va
+
+  %lo = bitcast <32 x i4> %res to <16 x i8>
+  store <16 x i8> %lo, ptr %po, align 16
+
+  %i.next = add nuw i64 %i, 1
+  %more = icmp ult i64 %i.next, %nvec
+  br i1 %more, label %loop, label %done
+
+done:
+  ret void
+}

bench/kernels/cmp_i4.ll

@@ -0,0 +1,46 @@
+; cmp_i4 -- unsigned per-field compare to a packed bitmask.
+;
+; NYB_DESC: <32 x i4> icmp ult -> packed <32 x i1> bitmask  (PARTIAL, see below)
+; NYB_OUT_BYTES_PER_VEC: 4
+;
+; PARTIAL RESULT -- read the number with the caveat. nybbler lowers the `icmp`
+; itself, but not the `bitcast <32 x i1> to <4 x i8>` that materializes its
+; result into a packed mask. That materialization is expensive in *both*
+; builds and dominates the loop, so the measured ratio understates the
+; compare lowering considerably. There is no way to fix this from the kernel
+; side: any consumer of a <K x i1> result (bitcast, sext, select) is outside
+; the set of instructions the pass rewrites. See the Limitations section of
+; docs/benchmarks.md.
+;
+; It is kept in the suite anyway because compare-to-bitmask is the actual
+; Parabix-style idiom, and an honest partial number is more useful than
+; silently omitting the compare lowering from the benchmark set.
+
+define void @nyb_kernel(ptr noalias %a, ptr noalias %b, ptr noalias %o, i64 %nvec) {
+entry:
+  %nonempty = icmp ugt i64 %nvec, 0
+  br i1 %nonempty, label %loop, label %done
+
+loop:
+  %i = phi i64 [ 0, %entry ], [ %i.next, %loop ]
+  %pa = getelementptr inbounds <16 x i8>, ptr %a, i64 %i
+  %pb = getelementptr inbounds <16 x i8>, ptr %b, i64 %i
+  %po = getelementptr inbounds <4 x i8>, ptr %o, i64 %i
+
+  %la = load <16 x i8>, ptr %pa, align 16
+  %lb = load <16 x i8>, ptr %pb, align 16
+  %va = bitcast <16 x i8> %la to <32 x i4>
+  %vb = bitcast <16 x i8> %lb to <32 x i4>
+
+  %lt = icmp ult <32 x i4> %va, %vb
+
+  %lo = bitcast <32 x i1> %lt to <4 x i8>
+  store <4 x i8> %lo, ptr %po, align 4
+
+  %i.next = add nuw i64 %i, 1
+  %more = icmp ult i64 %i.next, %nvec
+  br i1 %more, label %loop, label %done
+
+done:
+  ret void
+}