Skip to content

benchmarks.md

Latest commit

 

History

History
264 lines (205 loc) · 12.1 KB

benchmarks.md

File metadata and controls

264 lines (205 loc) · 12.1 KB
Raw

Benchmark guide

TensorLib includes a CPU benchmark harness for tensor kernels, autograd, neural-network modules, end-to-end training steps, and OpenMP scaling. This guide explains how to produce reproducible results and how to interpret the CSV output. It does not publish a permanent leaderboard: results depend on the CPU, compiler, build flags, thread placement, and system load.

Quick start

Run the short benchmark profile in a native-tuned release build:

make benchmark-quick BUILD_DIR=build-bench

The command builds bench_tensorlib, prints a report, and writes benchmark-results.csv. To choose the output file explicitly:

make benchmark-quick BUILD_DIR=build-bench \
  BENCHMARK_CSV=results.csv

Use a dedicated build directory. Benchmark builds enable TENSORLIB_NATIVE_OPTIMIZATIONS, which adds host-specific compiler flags and may produce binaries that do not run on another CPU.

Build and run without Make

The equivalent CMake workflow is:

cmake -S . -B build-bench -DCMAKE_BUILD_TYPE=Release \
  -DTENSORLIB_BUILD_BENCHMARKS=ON \
  -DTENSORLIB_NATIVE_OPTIMIZATIONS=ON
cmake --build build-bench --parallel
python3 scripts/run_benchmarks.py \
  --executable build-bench/bench_tensorlib \
  --profile quick --suite all --csv benchmark-results.csv

The Python runner adds the operating system, CPU model, logical processor count, and compiler to every CSV row. The C executable can also be invoked directly when host metadata is not needed:

build-bench/bench_tensorlib --profile quick --suite kernels \
  --csv raw-results.csv

Run build-bench/bench_tensorlib --help for the accepted command-line options.

Profiles

Profile Samples Minimum time per sample Warm-up Intended use
smoke 1 none none Correctness and portability checks only
quick 5 20 ms 20 ms Local development and regression triage
full 15 100 ms 250 ms Results intended for comparison or publication
fixed 5 none none Identical five-step training sequences across implementations

The quick/full harness first times one operation, then batches up to 1,000,000 iterations per sample to reach the profile's minimum duration. It reports time per operation. The median is the primary result; p95 is useful for spotting noisy runs. Do not treat smoke timings as performance measurements.

The fixed profile executes exactly five measured calls, with no calibration call or warm-up. Use it for matched training comparisons so both processes start from the same seeded state and perform the same optimizer/RNG sequence. Repeat in fresh processes; the five calls include the first-step packing cost.

For primary training timings use --suite training --profile fixed --no-diagnostics. This measures MNIST MLP (batch 64), TinyLM (batch 4), and CommandLM (batches 1 and 16) at each requested thread count. It disables phase clocks, allocation counters, and the extra backward profiling step. Run again without --no-diagnostics for a separate diagnostic profile; do not compare its throughput directly with the uninstrumented measurements.

The Python runner records the executable SHA-256, available CMake settings, CPU topology, inherited process affinity, thread environment, and checkout revision/status. Supply --source-revision REV from the clean source used for the build; without it the binary's source revision is explicitly unverified. The current checkout revision alone cannot identify an older executable.

Run the full profile with:

make benchmark-full BUILD_DIR=build-bench \
  BENCHMARK_CSV=results-full.csv

Suites and measurement boundaries

Select one suite with --suite kernels|autograd|nn|command|training|matmul|scaling|policy, or use all. Use training for the designated model acceptance workloads.

Suite Representative cases What the timed region includes
kernels allocation, views, copies, broadcasting, GELU, reductions, gather, matmul, RHS packing Public API call and output allocation/free
autograd composed elementwise forward; forward plus backward Dynamic graph construction; backward case also includes gradient computation, zeroing, and graph cleanup
nn Linear, LayerNorm, attention, decoder block, MNIST MLP, TinyLM Eager forward graph construction; training rows include loss, backward, optimizer step, and zero-grad
training MNIST MLP batch 64, TinyLM batch 4, CommandLM batches 1/16 Complete training calls at every requested thread count, including release and weight updates
matmul Linear forward, repacking, dinput/dweight, packing, narrow/tail shapes and attention gradients Multiplication or the explicitly named packing/copy/reduction boundary; reusable inputs outside timing
scaling large add, GELU, square matmul, TinyLM forward/train The same workload across the requested OpenMP thread ladder, plus derived speedup data for tensor kernels

The Python runner records TENSORLIB_* and OMP_* values in the CSV thread_environment field so differently configured runs can be identified.

Inputs and model construction happen outside the timed region. Most public tensor operations allocate their outputs, so output allocation and destruction remain inside it. Forward-only NN rows are labeled forward;graph-build: TensorLib has no no-grad execution mode, and these rows should not be described as inference-only latency.

Most kernels, autograd, and component nn cases use the first requested thread count. Use training or scaling for complete model ladders. In matmul, forward, forward-repack and direct dweight span the ladder; dinput, legacy dweight, packing, bias reduction and attention use its first count. Reductions and gather are serial today and identify that fact in the layout column.

forward_repack invalidates and rebuilds a weight pack every call; it does not perform an optimizer update. The training suite supplies that acceptance boundary. The isolated projection shapes use 4×128 input rows, including width 193/output 257 tails and outputs 17/7; they are not all CommandLM shapes.

Thread scaling

By default, the runner tests powers of two up to the number of logical processors, followed by the exact logical-processor count. Set a stable ladder explicitly when comparing runs:

make benchmark-scaling BUILD_DIR=build-bench \
  BENCHMARK_THREADS=1,2,4,8 \
  BENCHMARK_CSV=scaling.csv

Or run the executable directly:

build-bench/bench_tensorlib --suite scaling --profile full \
  --threads 1,2,4,8 --csv scaling-raw.csv

Speedup is baseline_time / current_time, where the baseline is the first thread count in the supplied ladder. For conventional one-thread-relative speedup, put 1 first. Parallel efficiency is speedup / thread_count.

When TensorLib is built without OpenMP, requests above one thread are written as skipped rows. Configure with -DTENSORLIB_ENABLE_OPENMP=OFF only when a serial portability result is intentional.

Optional OpenBLAS and PyTorch comparisons

Configure the benchmark build after OpenBLAS is installed. If CMake finds cblas.h and the OpenBLAS library, it creates bench_openblas. PyTorch is an optional runtime dependency of benchmarks/bench_pytorch.py; neither reference is required to build TensorLib.

Run the available references with a matched single-thread setting:

make benchmark-compare BUILD_DIR=build-bench \
  BENCHMARK_THREADS=1 BENCHMARK_REFERENCE_THREADS=1

This produces benchmark-results.csv, benchmark-results-openblas.csv, and/or benchmark-results-pytorch.csv, depending on what is installed.

Compare only rows with the same case, shape, profile, and actual thread count. The OpenBLAS baseline matches the three matmul workloads and includes output allocation/free. The PyTorch script matches those matmul shapes and the MNIST/TinyLM-style workloads in eager float32. These are workload references, not claims that framework internals, graph construction, memory allocators, or optimizer implementations are identical.

CSV schema

The native and reference runners share these columns:

Column Meaning
suite, case Workload group and stable case name
shape, layout Symbolic dimensions and relevant memory/execution layout
profile smoke, quick, full, or fixed
requested_threads, actual_threads Requested OpenMP/library setting and observed team size
median_seconds, p95_seconds Per-operation timing statistics
metric, value Unit and derived throughput, or milliseconds per call
iterations Operations batched into each timing sample
checksum Consumed result used to keep measured work observable
status ok, skipped, or derived

scripts/run_benchmarks.py appends os, cpu, logical_processors, and compiler. Scaling rows with status=derived contain speedup in value and parallel efficiency in checksum; their raw timing columns are zero. TinyLM scaling speedup is currently printed to the console but is not emitted as a separate derived CSV row, so calculate it from its status=ok timing rows when processing CSV data.

The fixed training profile also appears as profile=fixed. Diagnostic rows report storage allocation/live/peak bytes, requested auxiliary allocation counts and bytes for metadata/graphs/matmul scratch, and logical clone/QKV assembly copy bytes. Auxiliary reallocations count request sizes, not net live bytes. The live/peak figures cover tensor storage only. These counters exclude some NN/optimizer allocations, allocator overhead and other memory traffic. Backward per-operation byte deltas cover the operation callback; engine merges and reductions have separate timing rows. Run primary throughput separately with --no-diagnostics.

Metric formulas are conventional effective rates:

  • Matmul uses 2 * M * N * K / median_seconds FLOP/s, multiplied by batch dimensions when present.
  • Elementwise bandwidth counts the bytes named by the case (for example, two reads plus one write for contiguous binary operations).
  • Model throughput divides samples or tokens processed per call by the median.

Use these metrics to compare the same case. Effective GB/s is not a measurement of hardware memory-controller traffic, and GFLOP/s does not account for every instruction surrounding the nominal matmul work.

Reproducible reporting checklist

Before benchmarking, run the correctness suite:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel
ctest --test-dir build --output-on-failure

For results that others can reproduce:

  • Record the commit, CPU model, operating system, compiler, build type, CMake options, profile, suite, and thread ladder.
  • Close CPU-heavy applications, connect laptops to power, and use a stable performance governor where the operating system permits it.
  • Keep affinity and thread placement consistent, especially on hybrid CPUs.
  • Run the full profile in several fresh processes and report the distribution of medians rather than selecting the fastest run.
  • Compare actual_threads, not only the requested value, and give reference libraries the same thread count.
  • Retain the raw CSV files with any reported summary.

CI runs only the smoke profile. The project intentionally has no fixed performance threshold because shared CI hosts do not provide stable timing.

Adding a benchmark case

Place the case in the suite that owns its measurement boundary:

  • benchmarks/bench_kernels.c for public tensor primitives.
  • benchmarks/bench_autograd.c for graph lifecycle workloads.
  • benchmarks/bench_nn.c for layers or model-level workloads.
  • benchmarks/bench_scaling.c for explicit thread scaling.

Allocate reusable inputs outside the operation callback, consume at least one output value through the checksum, release per-call outputs inside the callback, and use the shared harness rather than adding a new timer. If a reference case is added, keep its shape and timed allocation boundary aligned and document any semantic difference. Validate harness changes with test_bench_harness and run the smoke benchmark before collecting timings.