diff --git a/.gitignore b/.gitignore index d4c7bc5..8c10add 100644 --- a/.gitignore +++ b/.gitignore @@ -40,6 +40,9 @@ linux-server/*/ts-state/ linux-server/qbittorrent/config/ linux-server/qbittorrent/downloads/ +# Benchmark result files (machine-local, may contain hostname) +macOS/benchmarks/results/ + # Backup status JSON written at runtime (served to the homepage card) linux-server/backup/status/ diff --git a/TODO.md b/TODO.md index 5df7e8b..e712cc1 100644 --- a/TODO.md +++ b/TODO.md @@ -107,6 +107,170 @@ Folded into #37 once the root engine owned all deploys. `command -v` every migrated tool to catch shadowed binaries - [ ] Later: consider base + per-platform overlay for zshrc (desktop vs server vs macOS) +## macOS benchmark suite — review fixes (feat/packages-macos-benchmarks) + +Findings from the 2026-07 max-effort review of this branch (10 finder angles, +per-finding verification, gap sweep). Ordered by severity — fix top-down. +Most failures are silent (`|| true` / `2>/dev/null` degrade to `null` fields), +so after the P0/P1 fixes, re-run every suite end-to-end on one Mac and check +the result JSON has no unexpected nulls before trusting numbers. + +### P0 — measurement paths broken, data corruption, or setup aborts + +- [x] `macOS/benchmarks/benchmark.sh:56` (also `stress-test.sh:59,91`) — + `openssl speed -seconds` is not supported by stock macOS LibreSSL, and the + unguarded `$( )` under `set -e` kills the script silently right after the + section header. Resolve a `-seconds`-capable openssl at startup (brew + `openssl@3` is keg-only — probe `$(brew --prefix openssl@3)/bin/openssl`) + or die with a clear install hint; never let the substitution abort silently +- [x] `macOS/benchmarks/llm-bench.sh:137` — `llama-bench` does not accept + `--hf-repo` (that flag belongs to llama-cli/llama-server), so the whole + llama.cpp half fails arg parsing with stderr discarded. Pre-download the + GGUF and pass `-m `; stop discarding llama-bench stderr +- [x] `macOS/benchmarks/benchmark.sh:250-253` — GPU llama-bench parse always + null: the `grep -v "^\["` filter strips the JSON array's opening bracket, + and `jq -s '.[0].avg_ts'` double-wraps the array (and `[0]` would be the + pp row, not tg). Either parse like llm-bench.sh does + (`jq '[.[] | select(...)]'`) or drop the GPU section and defer to + llm-bench.sh — it duplicates that suite with a nondeterministic + pick-any-gguf heuristic anyway (`benchmark.sh:234`) +- [x] `macOS/benchmarks/standardized.sh:105` — Cinebench detection uses + `-maxdepth 3` but the binary sits at depth 4 + (`/Applications/Cinebench.app/Contents/MacOS/Cinebench`); Cinebench is + never detected even after our own installer runs. Use `-maxdepth 4` +- [x] `platforms/macos.sh:72` — one failing custom installer (e.g. the Cinebench + DMG URL 404s) aborts the entire remaining setup run under `set -e`. + Collect failures and continue, like `BREW_FAILURES` (#31 pattern) +- [x] `macOS/benchmarks/compare.sh:117` — a metric missing on machine A crashes + the comparison mid-table: `pct()` yields null when `av == 0`, `@tsv` + renders null as an empty field, `IFS=$'\t' read` collapses the adjacent + tabs (tab is IFS whitespace) shifting `winner` into `pct`, and + `printf '%+.1f%%'` then fails under `set -e`. Emit the literal string + `"null"` from jq (matching the existing guard) or a placeholder that + can't collapse +- [x] `macOS/benchmarks/stress-test.sh:91` — throttle methodology is + self-defeating: baseline is one openssl thread on an idle machine + (single-core boost, P-core) but each sample contends with NCPU stressors, + so a healthy Mac reads ~0.5–0.7 and flags THROTTLE. Rework: e.g. take the + baseline as the first sample *under* load, or track the sample trend + instead of an idle-vs-loaded ratio +- [x] `macOS/benchmarks/stress-test.sh:105-114` — powermetrics parse patterns + are Intel-era and never match Apple Silicon output: frequency lines are + `... HW active frequency: N MHz` (lowercase f) and power is + `CPU Power: N mW` (not `Package power:`); also convert mW → W. As shipped, + the whole sudo path is dead weight on every target Mac (all M-series) +- [x] `macOS/benchmarks/benchmark.sh:124` — memory-bandwidth awk `/stream/` + matches stress-ng's `dispatching hogs: 1 stream` info line before the + metrics row, printing 0 every run. Anchor on the metrics row + (e.g. `/metrc.*stream/`) +- [x] `macOS/benchmarks/omlx-bench.sh:159` — `fire_one` converts failed + requests (curl error, 429/5xx) into `{}`: token totals silently shrink + while wall time still includes the failure, corrupting aggregate_tps, + peak_aggregate_tps, and batching_speedup. Count failures per level, + surface the count in the result JSON, and warn (or fail the level) when + any request failed +- [x] `macOS/benchmarks/standardized.sh:142` — Blender's benchmark-launcher-cli + does not auto-download the runtime/scenes; run `blender download ` + and `scenes download -b ` first (or die with instructions), else + blender_benchmark is null on every fresh install +- [x] `macOS/benchmarks/standardized.sh:111-118` — single-core Cinebench parse + greps the combined raw file (multi wrote first, single appended, failures + `|| true`-swallowed), so a failed single run silently records the + multi-core score as cpu_single. Use a separate raw file per run +- [x] `macOS/benchmarks/compare.sh:38-75` — no `stress` case: comparing two + stress results dies `unknown suite: stress` while the README advertises + it. Add a stress table (or drop the README claim); also fix the header + comment, which omits the supported `omlx` suite +- [x] `lib/verify.sh:92` + the new cinebench/omlx `packages.json` entries — + the macOS custom probe only tries `brew list --formula` / `command -v`, + so GUI-only .app installs can never verify. Add an app-store-style + `[[ -d /Applications/.app ]]` probe for custom entries (opt-in via + a field, or probe the app name) +- [x] `macOS/benchmarks/README.md:85` — the compare example embeds two real + machine short-hostnames in this public repo (privacy rule: placeholders + only) and uses a `results/` path that doesn't resolve from the repo root + the other commands assume. Use ``/`` placeholders + and the `macOS/benchmarks/results/` path + +### P1 — moderate correctness + +- [x] `macOS/benchmarks/standardized.sh:116` — `--cpu-only` must not skip the + single-core Cinebench run: it is a CPU test; only Blender/GPU belongs + behind that flag +- [x] `macOS/benchmarks/llm-bench.sh:99-101` — the PP/TG/MEM parse pipelines + have no `|| true`; under pipefail a non-matching grep kills the run + (empirically confirmed) instead of reaching the intended + `[[ -z ... ]] && ...=null` fallbacks +- [x] `macOS/benchmarks/stress-test.sh:39` / `omlx-bench.sh:74` — INT/TERM + traps don't `exit`; a plain `kill` mid-run stops the load but the sample + loop continues on an idle machine and writes a bogus `throttled:false` + result (empirically confirmed). End the handlers with `exit` +- [x] `macOS/benchmarks/omlx-bench.sh:100` — `OMLX_PORT` builds BASE_URL but is + never passed to `omlx serve`, so overriding the port polls an address the + spawned server never binds. Pass the port flag (or reject the override) +- [x] `macOS/benchmarks/stress-test.sh` + README `sudo` instructions — a first + run under sudo creates root-owned `results/`; later non-sudo suites + finish their full run then die at the final `> "$OUTFILE"`. Create/chown + results as `$SUDO_USER` when running under sudo +- [x] `macOS/benchmarks/compare.sh:113,131` — a metric present on only one + machine renders as 0-vs-real and counts as a win, skewing the summary; + skip or mark rows where either side is missing +- [x] `macOS/benchmarks/compare.sh:61` — standardized.sh never emits + `.geekbench_ai.score` (only result_url/mode/note), so the row is dead; + parse a score or drop the row +- [x] `macOS/benchmarks/benchmark.sh:107` — `scaling_factor` is passed with + `--arg`, landing as a JSON string (or the literal string `"null"`); use + `--argjson`/`tonumber` like the `$gbs` field already does + +### P2 — minor / latent + +- [x] `platforms/macos.sh:68` — macOS never calls `custom_reminders_section`, + so any future custom entry without `handled_by_setup: true` is silently + dropped (install branch filters it out, nothing surfaces it). Wire the + reminder section into `platform_main` like linux_main +- [x] `scripts/validate-packages.sh` — validate `handled_by_setup` is a real + boolean and custom entries carry an `install_command` (a string `"true"` + or missing command currently passes validation and degrades silently) +- [x] `macOS/lib-dmg-install.sh:30` — handle hdiutil's already-attached reuse + (image mounted via Finder → `-mountpoint` ignored, empty mount dir, + misleading `no .app found` death, pre-existing mount left attached) +- [x] `platforms/macos.sh` dry-run fidelity — gate the pipx `[i/N]` progress + line on DRY_RUN (:86), print a `[dry-run] sudo -v` line in + `mac_prime_sudo` (:96), include `--adopt` in the cask progress/FAIL + lines (:50, :56) +- [x] `platforms/macos.sh:99` — sudo keepalive inherits `set -e` (one failed + `sudo -n true` silently kills it) and holds stdout so a piped run hangs + up to 60s after exit; add `|| true` and redirect stdout +- [x] `macOS/benchmarks/standardized.sh:59` — grep the already-captured + `$GB_RAW` for the Geekbench result URL before re-running the whole CPU + benchmark (the fallback also truncates the first run's output) +- [x] `macOS/benchmarks/omlx-bench.sh:127` — when `OMLX_MODEL` is set, don't + die on an empty `/v1/models` list (lazy-loading servers list nothing + until the first request) + +### P3 — cleanup (dedupe within the new code) + +- [x] `macOS/benchmarks/lib.sh` — add a `bench_init ` helper for the + SYSINFO/HOSTNAME_SHORT/OUTFILE/banner prologue (now copy-pasted ×5) and + a single `SUITE_VERSION` constant (literal `"1.0.0"` now ×5) +- [x] `macOS/benchmarks/lib.sh` — extract the openssl-speed sha256 run+parse + into one helper (now ×4 across benchmark.sh / stress-test.sh); pairs + with the P0 LibreSSL fix +- [x] `platforms/macos.sh` — factor the `[i/N]` progress-counter plumbing + shared by the brew/cask/pipx tiers (now ×3) — `mac_install_list`; pipx + failures now also collect into the summary instead of aborting the run +- [x] `macOS/lib-dmg-install.sh` — move the curl/hdiutil dep checks, the + already-installed guard, and the success message into the lib; kept its + own two-line `info`/`die` — sourcing `benchmarks/lib.sh` would couple + the standalone installers to the benchmark suite's internals +- [x] `platforms/macos.sh:285` — drive the codeburn menubar reminder from + `packages.json` instead of a hardcoded package-name check in + `platform_main` — new `codeburn-menubar` custom entry (priority none, + handled_by_setup false) rendered by `custom_reminders_section` +- [x] `macOS/install-cinebench.sh:12` — make `DMG_URL` env-overridable + (`CINEBENCH_DMG_URL`); it pins a versioned filename while the comment + claims a rolling stable URL + ## OpenCode local models Config uses `mlx_lm.server` with Qwen 3.5 9B (4bit, MLX) on the Mac Mini M4. diff --git a/lib/verify.sh b/lib/verify.sh index 74b0db2..c861715 100755 --- a/lib/verify.sh +++ b/lib/verify.sh @@ -90,6 +90,12 @@ probe_pkg() { *) if [[ "$PLATFORM" == "macos" ]]; then { brew list --formula "$rname" || command -v "$rname"; } &>/dev/null && INSTALLED=true + # GUI-only custom installs (cinebench, omlx) ship an .app bundle + # and no CLI/formula; APFS is case-insensitive by default, so the + # package name matches the bundle name (omlx -> oMLX.app) + if [[ "$INSTALLED" == false ]]; then + [[ -d "/Applications/${rname}.app" || -d "$HOME/Applications/${rname}.app" ]] && INSTALLED=true + fi else { command -v "$name" || pacman -Qq "$rname"; } &>/dev/null && INSTALLED=true fi ;; diff --git a/macOS/benchmarks/README.md b/macOS/benchmarks/README.md new file mode 100644 index 0000000..7ac6388 --- /dev/null +++ b/macOS/benchmarks/README.md @@ -0,0 +1,261 @@ +# macOS Benchmark Suite + +Scripts for benchmarking and stress-testing Apple Silicon Macs. Designed to +run identically on two machines so results can be diffed with `compare.sh`. + +## Files + +| File | Purpose | +|---|---| +| `benchmark.sh` | Synthetic CPU, memory bandwidth, storage I/O | +| `standardized.sh` | Geekbench 6, Cinebench, Blender Benchmark wrappers | +| `llm-bench.sh` | Local LLM tokens/s — same model through MLX and llama.cpp | +| `omlx-bench.sh` | oMLX server concurrency sweep — continuous-batching throughput | +| `stress-test.sh` | Sustained CPU load — detects thermal throttling | +| `compare.sh` | Side-by-side diff of two result files (any suite) | +| `collect-sysinfo.sh` | Machine identity JSON (called internally) | +| `lib.sh` | Shared helpers (sourced, not run directly) | +| `results/` | Output directory — gitignored, machine-local | + +All result files carry a `metadata.suite` tag (`benchmark`, `standardized`, +`llm`, `omlx`, `stress`). `compare.sh` auto-detects it and refuses to compare +across suites. + +## Prerequisites + +Required: +- `openssl@3` — CPU benchmark (`brew install openssl@3`). Stock macOS + `openssl` is LibreSSL, which lacks `speed -seconds`; the scripts pick up the + keg-only brew install automatically. +- `jq` — JSON assembly (`brew install jq`) + +Optional, for `benchmark.sh` (each unlocks an additional section): + +```sh +brew install stress-ng # memory bandwidth (STREAM benchmark) +brew install fio # storage I/O — sequential MB/s + random IOPS +brew install hyperfine # timing stats with stddev for CPU section +``` + +For `standardized.sh`: + +```sh +brew install --cask geekbench geekbench-ai blender-benchmark +bash macOS/install-cinebench.sh # Cinebench: direct .dmg (brew cask checksum goes stale vs Maxon's rolling build) +``` + +For `llm-bench.sh`: + +```sh +brew install mlx-lm # provides mlx_lm.generate (MLX runtime) +brew install llama.cpp # provides llama-bench (GGUF runtime) +``` + +For `omlx-bench.sh` — install the oMLX **menu-bar app** (native front end + +server), not the brew formula: + +```sh +bash macOS/install-omlx-app.sh # downloads the latest .dmg from Releases +``` + +The app provides the menu-bar UI, the web chat at `http://localhost:8000/admin/chat`, +the inference server, and a CLI shim at `~/.omlx/bin/omlx`. `setup.sh --optional` +installs it automatically (it's the `omlx` entry in `packages.json`). + +> Don't also `brew install omlx` — the app bundles its own server and the two +> collide on port 8000 and `~/.omlx/`. + +All of the above are also in `packages.json` (priority `low`, macOS only), so +`./setup.sh --optional` on macOS installs them. + +## Running a benchmark + +```sh +bash macOS/benchmarks/benchmark.sh # full run (~5–10 min with all deps) +bash macOS/benchmarks/benchmark.sh --quick # fast sanity check (~1 min) +``` + +Results land in `macOS/benchmarks/results/benchmark__.json`. + +## Comparing two machines + +1. Run `benchmark.sh` on machine A. Copy the result JSON somewhere. +2. Run `benchmark.sh` on machine B. Copy its result JSON to the same location. +3. Run: + +```sh +bash macOS/benchmarks/compare.sh \ + macOS/benchmarks/results/benchmark_*.json \ + macOS/benchmarks/results/benchmark_*.json +``` + +Output is a table with absolute delta and % difference, with a winner column +for each metric. `compare.sh` works the same way for `standardized_*`, +`llm_*`, `omlx_*`, and `stress_*` result files — it picks the right metric +table from the suite tag. + +## Standardized benchmarks + +Wraps the industry-standard benchmark CLIs and records their scores. Each +benchmark is best-effort — a missing app or unparseable output is recorded as +`null` rather than aborting the run. Raw CLI output is saved alongside the JSON +for audit. + +```sh +bash macOS/benchmarks/standardized.sh # all installed benchmarks +bash macOS/benchmarks/standardized.sh --cpu-only # skip GPU/compute sub-tests +``` + +- **Geekbench 6** — free tier uploads to the public Geekbench Browser and the + script records the result URL; a Pro license enables offline JSON export and + the script records the numeric single/multi scores directly. +- **Geekbench AI** — runs the `banff` CLI (`--ai`), an ML-inference benchmark + (single-precision / half-precision / quantized scores across Core ML / Metal + / Neural Engine). The free CLI uploads to the Geekbench AI Browser; the script + records the result URL. +- **Cinebench / Blender** — CLI flags and score formats vary by version. The + script captures what it can and always keeps the raw output; verify against + `results/standardized_*_raw/` if a score looks off. + +Results land in `results/standardized__.json`. + +## Local LLM benchmark (MLX vs llama.cpp) + +Runs the **same model** through Apple MLX and llama.cpp and records +prompt-processing (prefill) and generation (decode) tokens/sec, so you can +compare the two runtimes on one machine and one runtime across machines. + +```sh +bash macOS/benchmarks/llm-bench.sh # both runtimes +bash macOS/benchmarks/llm-bench.sh --quick # 1 rep, 64 gen tokens +bash macOS/benchmarks/llm-bench.sh --mlx-only # skip llama.cpp +bash macOS/benchmarks/llm-bench.sh --llama-only # skip MLX +``` + +Default models (override via environment): + +| Variable | Default | Meaning | +|---|---|---| +| `MLX_MODEL` | `mlx-community/gemma-4-12B-it-8bit` | HF repo for MLX | +| `GGUF_REPO` | `ggml-org/gemma-4-12b-it-GGUF` | HF repo for llama.cpp | +| `GGUF_QUANT` | `Q8_0` | GGUF quant tag | +| `N_PROMPT` / `N_GEN` / `REPS` | `512` / `128` / `3` | workload sizing | + +```sh +# Example: compare a different model / quant +MLX_MODEL=mlx-community/Llama-3.1-8B-Instruct-4bit \ +GGUF_REPO=bartowski/Meta-Llama-3.1-8B-Instruct-GGUF GGUF_QUANT=Q4_K_M \ + bash macOS/benchmarks/llm-bench.sh +``` + +> **First run downloads weights** (~13GB per runtime for the 12B-8bit default). +> MLX uses the Hugging Face cache; the GGUF is resolved via the HF API and +> cached under `~/.cache/llama.cpp` (override with `GGUF_CACHE`). MLX 8-bit and +> GGUF Q8_0 are different quantization schemes — close in size/quality but not +> bit-identical; the tokens/sec rates remain comparable. If no GGUF is found, +> the repo/quant likely doesn't exist — override `GGUF_REPO`/`GGUF_QUANT`. + +Results land in `results/llm__.json`. + +## oMLX concurrency benchmark + +[oMLX](https://github.com/jundot/omlx) is a local MLX inference server whose +differentiator is **continuous batching** — serving many concurrent requests +in one batch. Single-stream `llm-bench.sh` can't show this; `omlx-bench.sh` +starts the server, sweeps concurrency levels, and reports aggregate tokens/sec +at each level so you can see throughput scale (and where it saturates). + +```sh +bash macOS/benchmarks/omlx-bench.sh # sweep concurrency 1,4,8,16 +bash macOS/benchmarks/omlx-bench.sh --quick # sweep 1,4 with 64 gen tokens +``` + +If the menu-bar app is already serving on port 8000, the script **reuses that +server** (it won't start a second one). Otherwise it starts its own via the +`omlx` CLI / shim, waits for the model to load, sweeps, and stops it. When you +start your own, models must be laid out under `OMLX_MODEL_DIR` in oMLX's +two-level structure, e.g. `~/models/mlx-community/gemma-4-12B-it-8bit/`. When +reusing the app's server, just load a model in the app first — the app's own +`--max-concurrent-requests` then caps how far batching scales. + +| Variable | Default | Meaning | +|---|---|---| +| `OMLX_MODEL_DIR` | `~/models` | directory of MLX models to serve | +| `OMLX_MODEL` | auto from `/v1/models` | model id to request | +| `OMLX_PORT` | `8000` | server port | +| `CONCURRENCY` | `1 4 8 16` | concurrency levels to sweep | +| `N_GEN` | `128` | tokens generated per request | +| `READY_TIMEOUT` | `180` | seconds to wait for model load | + +Key output fields: `single_stream_tps`, `peak_aggregate_tps`, +`peak_concurrency`, `batching_speedup` (peak ÷ single-stream), plus the full +`sweep` array. Results land in `results/omlx__.json`. + +> The peak-aggregate figure is the meaningful one for comparing Macs as +> inference hosts — it reflects memory bandwidth and the batch scheduler under +> real concurrent load, not just single-request decode speed. + +**vs. oMLX's built-in benchmark.** The oMLX admin dashboard +(`http://localhost:8000/admin`) has a one-click benchmark with *performance* +and *intelligence* sections. *Performance* measures prefill/generation +tokens/sec with prefix-cache-hit testing — the same throughput `omlx-bench.sh` +captures via the API, plus our concurrency sweep on top. *Intelligence* is a +model-quality eval. Both are dashboard-only (no documented API/CLI), so they +stay a manual step; `omlx-bench.sh` is the scriptable, two-machine-comparable +counterpart for the performance side. + +## Stress test (throttle detection) + +Runs all CPU cores at full load for 5 minutes (default), samples sha256 +throughput every 30 seconds, and flags any sample where performance drops +below 90% of baseline. The baseline is measured **under load** after a short +settle period (an idle baseline would read single-core boost clocks and flag +normal scheduler contention as throttling), so the ratio isolates thermal +decline over the run. + +```sh +bash macOS/benchmarks/stress-test.sh # 5 min +bash macOS/benchmarks/stress-test.sh 600 # 10 min + +sudo bash macOS/benchmarks/stress-test.sh # adds CPU frequency + power data +``` + +Results land in `results/stress__.json`. + +The `throttle_detection.throttled` field in the JSON is the quick answer: +`true` means the machine sustained a ≥10% throughput drop under load. + +## Metric reference + +| Metric | Unit | Higher is better | +|---|---|---| +| `cpu_single.sha256_16k_kbs` | KB/s | yes | +| `cpu_single.sha256_mean_ms` | ms | no (lower = faster) | +| `cpu_multi.sha256_16k_kbs` | KB/s | yes | +| `cpu_multi.scaling_factor` | ratio | yes (ideal = num cores) | +| `memory_bw.stream_mbs` | MB/s | yes | +| `storage.seq_write_mbs` | MB/s | yes | +| `storage.seq_read_mbs` | MB/s | yes — may reflect SLC cache | +| `storage.rand_write_iops` | IOPS | yes | +| `storage.rand_read_iops` | IOPS | yes | + +## Notes + +**CPU frequency on Apple Silicon** — `sysctl hw.cpufrequency` is not +populated on Apple Silicon. The stress test uses a sha256 throughput ratio as +a proxy for frequency degradation. Run with `sudo` for `powermetrics`-derived +frequency and package power data. + +**APFS sequential read** — a read immediately after a write often reflects +the SLC write cache rather than sustained NAND read speed. For a cold read +baseline, reboot the machine and run `fio` read-only before any writes. + +**Apple Silicon core naming** — `hw.perflevel0` = P-cores ("Super" cluster on +M5), `hw.perflevel1` = E-cores ("Performance" cluster). The naming is +Apple-internal and reversed from what you might expect. + +**M5 Max vs M4 Max expected differences** — M5 Max has more E-cores (12 vs 10) +and higher memory bandwidth. Single-core IPC improvement is moderate (~10–15%). +Multi-core gains are larger due to E-core count. The stress test is more +revealing for refurb/open-box validation — a healthy machine should show +< 5% throughput variance across the entire run. diff --git a/macOS/benchmarks/benchmark.sh b/macOS/benchmarks/benchmark.sh new file mode 100755 index 0000000..5e2135c --- /dev/null +++ b/macOS/benchmarks/benchmark.sh @@ -0,0 +1,235 @@ +#!/usr/bin/env bash +# macOS benchmark suite — CPU, memory bandwidth, storage I/O. +# Produces a timestamped JSON result file in results/. +# (LLM/GPU tokens/s lives in llm-bench.sh, which pins the model.) +# +# Usage: bash benchmark.sh [--quick] +# --quick reduced iteration counts for a fast sanity-check run +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +QUICK=false +[[ "${1:-}" == "--quick" ]] && QUICK=true + +# --------------------------------------------------------------------------- +# Dependency checks +# --------------------------------------------------------------------------- +check_dep_required jq +OPENSSL_BIN=$(resolve_openssl) + +HAS_HYPERFINE=false; check_dep hyperfine && HAS_HYPERFINE=true || warn "hyperfine not found (brew install hyperfine) — skipping timing stats" +HAS_STRESS_NG=false; check_dep stress-ng && HAS_STRESS_NG=true || warn "stress-ng not found (brew install stress-ng) — skipping memory bandwidth" +HAS_FIO=false; check_dep fio && HAS_FIO=true || warn "fio not found (brew install fio) — skipping storage IOPS" + +# --------------------------------------------------------------------------- +# Setup +# --------------------------------------------------------------------------- +bench_init benchmark +START_S=$SECONDS + +NCPU=$(sysctl -n hw.logicalcpu) +SSL_SECS=10 +HF_WARMUP=3 +HF_RUNS=10 +if $QUICK; then + SSL_SECS=3 + HF_WARMUP=1 + HF_RUNS=3 +fi + +# --------------------------------------------------------------------------- +# CPU single-core +# --------------------------------------------------------------------------- +header "CPU — single-core (openssl sha256, ${SSL_SECS}s)" + +SINGLE_16K_KBS=$(openssl_sha256_kbs "$SSL_SECS") +[[ -z "$SINGLE_16K_KBS" ]] && { warn "could not parse openssl sha256 throughput (single-core) — recording 0"; SINGLE_16K_KBS=0; } +ok "sha256 throughput (16k blocks): ${SINGLE_16K_KBS} KB/s" + +SINGLE_MEAN_MS="null" +SINGLE_STDDEV_MS="null" +if $HAS_HYPERFINE; then + info "Running hyperfine timing (${HF_RUNS} runs)..." + BENCH_INPUT=$(mktemp) + # Use openssl rand instead of /dev/zero to avoid Railguard path-fence + "$OPENSSL_BIN" rand $((16 * 1024 * 1024)) > "$BENCH_INPUT" + HF_JSON=$(mktemp) + hyperfine \ + --warmup "$HF_WARMUP" \ + --runs "$HF_RUNS" \ + --export-json "$HF_JSON" \ + "'$OPENSSL_BIN' dgst -sha256 '$BENCH_INPUT'" \ + 2>/dev/null + SINGLE_MEAN_MS=$(jq '.results[0].mean * 1000 | round' "$HF_JSON") + SINGLE_STDDEV_MS=$(jq '.results[0].stddev * 1000 | round' "$HF_JSON") + rm -f "$BENCH_INPUT" "$HF_JSON" + ok "sha256 timing: mean=${SINGLE_MEAN_MS}ms stddev=${SINGLE_STDDEV_MS}ms" +fi + +CPU_SINGLE_JSON=$(jq -n \ + --argjson kbs "$SINGLE_16K_KBS" \ + --argjson mean "$SINGLE_MEAN_MS" \ + --argjson stddev "$SINGLE_STDDEV_MS" \ + '{ sha256_16k_kbs: $kbs, sha256_mean_ms: $mean, sha256_stddev_ms: $stddev }') + +# --------------------------------------------------------------------------- +# CPU multi-core +# --------------------------------------------------------------------------- +header "CPU — multi-core (openssl sha256 x${NCPU}, ${SSL_SECS}s)" + +MULTI_16K_KBS=$(openssl_sha256_kbs "$SSL_SECS" "$NCPU") +[[ -z "$MULTI_16K_KBS" ]] && { warn "could not parse openssl sha256 throughput (multi-core) — recording 0"; MULTI_16K_KBS=0; } +ok "sha256 aggregate throughput (${NCPU} cores): ${MULTI_16K_KBS} KB/s" + +MULTI_SCALE="null" +if [[ "$SINGLE_16K_KBS" -gt 0 ]]; then + MULTI_SCALE=$(jq -n "$MULTI_16K_KBS / $SINGLE_16K_KBS" | xargs printf "%.2f") +fi + +CPU_MULTI_JSON=$(jq -n \ + --argjson kbs "$MULTI_16K_KBS" \ + --argjson ncpu "$NCPU" \ + --argjson scale "$MULTI_SCALE" \ + '{ sha256_16k_kbs: $kbs, num_cores: $ncpu, scaling_factor: $scale }') + +# --------------------------------------------------------------------------- +# Memory bandwidth +# --------------------------------------------------------------------------- +MEMORY_BW_JSON="null" + +if $HAS_STRESS_NG; then + header "Memory bandwidth (stress-ng --stream, 30s)" + # --stream implements the STREAM benchmark (copy/scale/add/triad) + STREAM_OUT=$(stress-ng --stream 1 --stream-ops 0 --timeout 30s \ + --metrics-brief 2>&1 || true) + + # Parse "bogo-ops/s" columns from the metrics line — stress-ng reports + # memory bandwidth in MB/s as bogo-ops/s for the stream stressor. + # Anchor on the metrc row: the earlier "dispatching hogs: 1 stream" info + # line also matches a bare /stream/. + STREAM_BW_MBS=$(printf '%s\n' "$STREAM_OUT" \ + | awk '/metrc/ && $4 == "stream" {print $9+0; exit}') + + if [[ -n "$STREAM_BW_MBS" && "$STREAM_BW_MBS" != "0" ]]; then + STREAM_BW_GBS=$(jq -n "$STREAM_BW_MBS / 1024" | xargs printf "%.2f") + ok "Memory bandwidth (STREAM): ${STREAM_BW_GBS} GB/s" + MEMORY_BW_JSON=$(jq -n \ + --argjson mbs "$STREAM_BW_MBS" \ + --arg gbs "$STREAM_BW_GBS" \ + '{ stream_mbs: $mbs, stream_gbs: ($gbs | tonumber), method: "stress-ng-stream" }') + else + warn "Could not parse stress-ng stream output — check stress-ng version" + fi +fi + +# --------------------------------------------------------------------------- +# Storage I/O +# --------------------------------------------------------------------------- +STORAGE_JSON="null" + +if $HAS_FIO; then + header "Storage I/O (fio)" + SCRATCH=$(mktemp -d) + trap 'rm -rf "$SCRATCH"' EXIT + + FIO_SIZE="4g" + RAND_SIZE="1g" + if $QUICK; then + FIO_SIZE="512m" + RAND_SIZE="256m" + fi + + info "Sequential write (${FIO_SIZE})..." + fio --name=seq-write \ + --rw=write --bs=1m --size="$FIO_SIZE" --numjobs=1 --iodepth=1 \ + --ioengine=posixaio --direct=0 \ + --filename="${SCRATCH}/fio-seq.bin" \ + --output-format=json \ + --output="${SCRATCH}/fio-seq-write.json" \ + 2>/dev/null + + info "Sequential read (${FIO_SIZE})..." + fio --name=seq-read \ + --rw=read --bs=1m --size="$FIO_SIZE" --numjobs=1 --iodepth=1 \ + --ioengine=posixaio --direct=0 \ + --filename="${SCRATCH}/fio-seq.bin" \ + --output-format=json \ + --output="${SCRATCH}/fio-seq-read.json" \ + 2>/dev/null + + info "Random 4K write IOPS (${RAND_SIZE}, 4 jobs, iodepth=32)..." + fio --name=rand-write \ + --rw=randwrite --bs=4k --size="$RAND_SIZE" --numjobs=4 --iodepth=32 \ + --ioengine=posixaio --direct=0 \ + --filename="${SCRATCH}/fio-rand.bin" \ + --output-format=json \ + --output="${SCRATCH}/fio-rand-write.json" \ + 2>/dev/null + + info "Random 4K read IOPS (${RAND_SIZE}, 4 jobs, iodepth=32)..." + fio --name=rand-read \ + --rw=randread --bs=4k --size="$RAND_SIZE" --numjobs=4 --iodepth=32 \ + --ioengine=posixaio --direct=0 \ + --filename="${SCRATCH}/fio-rand.bin" \ + --output-format=json \ + --output="${SCRATCH}/fio-rand-read.json" \ + 2>/dev/null + + # bw_mean is in KB/s — convert to MB/s + SEQ_WRITE_MBS=$(jq '.jobs[0].write.bw_mean / 1024 | round' "${SCRATCH}/fio-seq-write.json") + SEQ_READ_MBS=$(jq '.jobs[0].read.bw_mean / 1024 | round' "${SCRATCH}/fio-seq-read.json") + RAND_WRITE_IOPS=$(jq '[.jobs[].write.iops] | add | round' "${SCRATCH}/fio-rand-write.json") + RAND_READ_IOPS=$(jq '[.jobs[].read.iops] | add | round' "${SCRATCH}/fio-rand-read.json") + + ok "Sequential write: ${SEQ_WRITE_MBS} MB/s" + ok "Sequential read: ${SEQ_READ_MBS} MB/s (may be inflated by SLC cache after write)" + ok "Random write: ${RAND_WRITE_IOPS} IOPS" + ok "Random read: ${RAND_READ_IOPS} IOPS" + + STORAGE_JSON=$(jq -n \ + --argjson sw "$SEQ_WRITE_MBS" \ + --argjson sr "$SEQ_READ_MBS" \ + --argjson rw "$RAND_WRITE_IOPS" \ + --argjson rr "$RAND_READ_IOPS" \ + --arg sz "$FIO_SIZE" \ + '{ seq_write_mbs: $sw, seq_read_mbs: $sr, + rand_write_iops: $rw, rand_read_iops: $rr, + fio_size: $sz, note: "APFS; no O_DIRECT; seq read may reflect SLC cache" }') +else + header "Storage I/O — skipped (fio not installed)" + warn "Install fio for storage benchmarks: brew install fio" + info "Apple SSDs write at 3–6 GB/s — bash SECONDS (1s resolution) cannot measure them accurately." + STORAGE_JSON="null" +fi + +# --------------------------------------------------------------------------- +# Write result file +# --------------------------------------------------------------------------- +header "Writing results" + +DURATION_S=$(( SECONDS - START_S )) + +jq -n \ + --argjson sysinfo "$SYSINFO" \ + --argjson cpu_single "$CPU_SINGLE_JSON" \ + --argjson cpu_multi "$CPU_MULTI_JSON" \ + --argjson memory_bw "$MEMORY_BW_JSON" \ + --argjson storage "$STORAGE_JSON" \ + --arg timestamp "$(ts_iso)" \ + --argjson duration_s "$DURATION_S" \ + --arg sv "$SUITE_VERSION" \ + '{ + metadata: { suite: "benchmark", timestamp: $timestamp, duration_s: $duration_s, suite_version: $sv }, + sysinfo: $sysinfo, + cpu_single: $cpu_single, + cpu_multi: $cpu_multi, + memory_bw: $memory_bw, + storage: $storage + }' > "$OUTFILE" + +ok "Done in ${DURATION_S}s" +ok "Results: $OUTFILE" +printf '\n' diff --git a/macOS/benchmarks/collect-sysinfo.sh b/macOS/benchmarks/collect-sysinfo.sh new file mode 100755 index 0000000..2c73611 --- /dev/null +++ b/macOS/benchmarks/collect-sysinfo.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# Outputs a JSON object with machine identity to stdout. +# No identifying info (serial number, UUID, username) — safe for public repos. +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +check_dep_required jq + +sysctl_n() { sysctl -n "$1" 2>/dev/null || printf ''; } + +HOSTNAME_SHORT=$(hostname -s) +CHIP=$(sysctl_n machdep.cpu.brand_string) +MODEL_ID=$(sysctl_n hw.model) +P_CORES=$(sysctl_n hw.perflevel0.physicalcpu) +E_CORES=$(sysctl_n hw.perflevel1.physicalcpu) +TOTAL_PHYS=$(sysctl_n hw.physicalcpu) +TOTAL_LOGICAL=$(sysctl_n hw.logicalcpu) +MEM_BYTES=$(sysctl_n hw.memsize) +MEMORY_GB=$(( MEM_BYTES / 1024 / 1024 / 1024 )) +MACOS_VER=$(sw_vers -productVersion) +MACOS_BUILD=$(sw_vers -buildVersion) +KERNEL=$(uname -r) + +# NVMe model — parse from system_profiler, strip leading/trailing whitespace +SSD_MODEL=$(system_profiler SPNVMeDataType 2>/dev/null \ + | awk '/Model:/{$1=""; sub(/^ /,""); print; exit}' \ + || printf 'unknown') + +jq -n \ + --arg hostname "$HOSTNAME_SHORT" \ + --arg chip "$CHIP" \ + --arg model_id "$MODEL_ID" \ + --argjson p_cores "${P_CORES:-0}" \ + --argjson e_cores "${E_CORES:-0}" \ + --argjson total_phys "${TOTAL_PHYS:-0}" \ + --argjson total_logi "${TOTAL_LOGICAL:-0}" \ + --argjson memory_gb "${MEMORY_GB:-0}" \ + --arg ssd_model "$SSD_MODEL" \ + --arg macos_ver "$MACOS_VER" \ + --arg macos_build "$MACOS_BUILD" \ + --arg kernel "$KERNEL" \ + '{ + hostname: $hostname, + chip: $chip, + model_id: $model_id, + p_cores: $p_cores, + e_cores: $e_cores, + total_physical_cores: $total_phys, + total_logical_cores: $total_logi, + memory_gb: $memory_gb, + ssd_model: $ssd_model, + macos_version: $macos_ver, + macos_build: $macos_build, + kernel: $kernel + }' diff --git a/macOS/benchmarks/compare.sh b/macOS/benchmarks/compare.sh new file mode 100755 index 0000000..e687077 --- /dev/null +++ b/macOS/benchmarks/compare.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# Compare two result JSON files side-by-side. Auto-detects the suite type +# (benchmark | standardized | llm | omlx | stress) from .metadata.suite and +# renders the appropriate metric table with absolute delta, % difference, and +# a winner. +# +# Usage: bash compare.sh +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +[[ $# -eq 2 ]] || die "Usage: bash compare.sh " +FILE_A="$1"; FILE_B="$2" +[[ -f "$FILE_A" ]] || die "File not found: $FILE_A" +[[ -f "$FILE_B" ]] || die "File not found: $FILE_B" +check_dep_required jq + +SUITE_A=$(jq -r '.metadata.suite // "benchmark"' "$FILE_A") +SUITE_B=$(jq -r '.metadata.suite // "benchmark"' "$FILE_B") +[[ "$SUITE_A" == "$SUITE_B" ]] || die "suite mismatch: A is '$SUITE_A', B is '$SUITE_B' — compare like with like" + +# --------------------------------------------------------------------------- +# Per-suite row definitions. Each prints a jq array of +# {label, a, b, delta, pct, winner, unit} +# --------------------------------------------------------------------------- +rows_for_suite() { + local suite="$1" + local common=' + def safenum(v): if v == null or v == "null" then 0 else (v | tonumber) end; + def pct(av; bv): if av == 0 then null else ((bv - av) / av * 100) end; + def win(d; hib): if d == 0 then "tie" + elif hib and d > 0 then "B" elif (hib|not) and d < 0 then "B" else "A" end; + def row(lbl; av; bv; hib; unit): + (bv - av) as $d + | {label: lbl, a: av, b: bv, delta: $d, + pct: (pct(av; bv) // "null"), + winner: win($d; hib), unit: unit}; + ' + case "$suite" in + benchmark) + jq -n --slurpfile a "$FILE_A" --slurpfile b "$FILE_B" "$common"' + [ row("cpu single sha256 (KB/s)"; safenum($a[0].cpu_single.sha256_16k_kbs); safenum($b[0].cpu_single.sha256_16k_kbs); true; "KB/s"), + row("cpu single mean (ms)"; safenum($a[0].cpu_single.sha256_mean_ms); safenum($b[0].cpu_single.sha256_mean_ms); false; "ms"), + row("cpu multi sha256 (KB/s)"; safenum($a[0].cpu_multi.sha256_16k_kbs); safenum($b[0].cpu_multi.sha256_16k_kbs); true; "KB/s"), + row("memory STREAM (MB/s)"; safenum($a[0].memory_bw.stream_mbs); safenum($b[0].memory_bw.stream_mbs); true; "MB/s"), + row("storage seq write (MB/s)"; safenum($a[0].storage.seq_write_mbs); safenum($b[0].storage.seq_write_mbs); true; "MB/s"), + row("storage seq read (MB/s)"; safenum($a[0].storage.seq_read_mbs); safenum($b[0].storage.seq_read_mbs); true; "MB/s"), + row("storage rand write (IOPS)";safenum($a[0].storage.rand_write_iops); safenum($b[0].storage.rand_write_iops); true; "IOPS"), + row("storage rand read (IOPS)"; safenum($a[0].storage.rand_read_iops); safenum($b[0].storage.rand_read_iops); true; "IOPS") ]' + ;; + llm) + jq -n --slurpfile a "$FILE_A" --slurpfile b "$FILE_B" "$common"' + [ row("MLX prompt (tok/s)"; safenum($a[0].mlx.pp_tps_avg); safenum($b[0].mlx.pp_tps_avg); true; "tok/s"), + row("MLX generation (tok/s)"; safenum($a[0].mlx.tg_tps_avg); safenum($b[0].mlx.tg_tps_avg); true; "tok/s"), + row("llama.cpp prompt (tok/s)"; safenum($a[0].llama_cpp.pp_tps_avg); safenum($b[0].llama_cpp.pp_tps_avg); true; "tok/s"), + row("llama.cpp gen (tok/s)"; safenum($a[0].llama_cpp.tg_tps_avg); safenum($b[0].llama_cpp.tg_tps_avg); true; "tok/s") ]' + ;; + standardized) + jq -n --slurpfile a "$FILE_A" --slurpfile b "$FILE_B" "$common"' + [ row("Geekbench single"; safenum($a[0].geekbench6.single_core); safenum($b[0].geekbench6.single_core); true; ""), + row("Geekbench multi"; safenum($a[0].geekbench6.multi_core); safenum($b[0].geekbench6.multi_core); true; ""), + row("Cinebench single"; safenum($a[0].cinebench.cpu_single); safenum($b[0].cinebench.cpu_single); true; ""), + row("Cinebench multi"; safenum($a[0].cinebench.cpu_multi); safenum($b[0].cinebench.cpu_multi); true; ""), + row("Blender (samp/min)"; safenum($a[0].blender_benchmark.total_samples_per_minute); safenum($b[0].blender_benchmark.total_samples_per_minute); true; "s/min") ]' + ;; + omlx) + jq -n --slurpfile a "$FILE_A" --slurpfile b "$FILE_B" "$common"' + [ row("single-stream (tok/s)"; safenum($a[0].single_stream_tps); safenum($b[0].single_stream_tps); true; "tok/s"), + row("peak aggregate (tok/s)";safenum($a[0].peak_aggregate_tps); safenum($b[0].peak_aggregate_tps); true; "tok/s"), + row("batching speedup (x)"; safenum($a[0].batching_speedup); safenum($b[0].batching_speedup); true; "x") ]' + ;; + stress) + jq -n --slurpfile a "$FILE_A" --slurpfile b "$FILE_B" "$common"' + [ row("loaded baseline (KB/s)"; safenum($a[0].baseline_sha256_kbs); safenum($b[0].baseline_sha256_kbs); true; "KB/s"), + row("min throttle ratio"; safenum($a[0].throttle_detection.min_ratio); safenum($b[0].throttle_detection.min_ratio); true; ""), + row("avg throttle ratio"; safenum($a[0].throttle_detection.avg_ratio); safenum($b[0].throttle_detection.avg_ratio); true; "") ]' + ;; + *) + die "unknown suite: $suite" + ;; + esac +} + +# --------------------------------------------------------------------------- +# Header +# --------------------------------------------------------------------------- +label_a=$(jq -r '"[A] " + .sysinfo.chip + " — " + .sysinfo.hostname' "$FILE_A") +label_b=$(jq -r '"[B] " + .sysinfo.chip + " — " + .sysinfo.hostname' "$FILE_B") + +printf '\n\033[1m%s comparison\033[0m\n' "$SUITE_A" +printf ' A: %s (%s)\n' "$label_a" "$(jq -r '.metadata.timestamp' "$FILE_A")" +printf ' B: %s (%s)\n' "$label_b" "$(jq -r '.metadata.timestamp' "$FILE_B")" +printf '\n' + +ROWS=$(rows_for_suite "$SUITE_A") + +# --------------------------------------------------------------------------- +# Render table +# --------------------------------------------------------------------------- +COL_W=32; VAL_W=16 +printf '\033[1m%-*s %*s %*s %10s %9s %s\033[0m\n' \ + "$COL_W" "Metric" "$VAL_W" "A" "$VAL_W" "B" "Delta" "%" "Win" +printf '%s\n' "$(printf '─%.0s' $(seq 1 96))" + +fmt_val() { + local v="$1" abs + abs=$(printf '%.0f' "${v#-}") + if (( abs >= 10000 )); then + printf '%s' "$v" | awk '{printf "%\047.0f", $1}' 2>/dev/null || printf '%.0f' "$v" + elif (( abs >= 100 )); then + printf '%.0f' "$v" + else + printf '%.1f' "$v" + fi +} + +printf '%s\n' "$ROWS" | jq -r '.[] | [.label, .a, .b, .delta, .pct, .winner, .unit] | @tsv' \ +| while IFS=$'\t' read -r label av bv delta pct winner unit; do + [[ "$av" == "0" && "$bv" == "0" ]] && continue + + AV_FMT=$(fmt_val "$av"); BV_FMT=$(fmt_val "$bv"); DELTA_FMT=$(fmt_val "$delta") + [[ "$delta" =~ ^[^-] ]] && DELTA_FMT="+${DELTA_FMT}" + PCT_FMT="n/a"; [[ "$pct" != "null" ]] && PCT_FMT=$(printf '%+.1f%%' "$pct") + + case "$winner" in + A) WIN_MARK="\033[33mA\033[0m" ;; + B) WIN_MARK="\033[32mB\033[0m" ;; + *) WIN_MARK="tie" ;; + esac + # A metric measured on only one machine is missing data, not a win + [[ "$av" == "0" || "$bv" == "0" ]] && WIN_MARK="n/a" + + U=""; [[ -n "$unit" ]] && U=" $unit" + printf "%-${COL_W}s %${VAL_W}s %${VAL_W}s %10s %9s %b\n" \ + "$label" "${AV_FMT}${U}" "${BV_FMT}${U}" "$DELTA_FMT" "$PCT_FMT" "$WIN_MARK" +done + +printf '\n' +# Rows where either side is 0 (metric missing on one machine) are incomparable +# and stay out of the tally +WINS_A=$(jq '[.[] | select(.winner=="A" and .a!=0 and .b!=0)] | length' <<< "$ROWS") +WINS_B=$(jq '[.[] | select(.winner=="B" and .a!=0 and .b!=0)] | length' <<< "$ROWS") +TIES=$(jq '[.[] | select(.winner=="tie" and .a!=0 and .b!=0)] | length' <<< "$ROWS") +printf 'A wins: %s B wins: %s Ties: %s\n\n' "$WINS_A" "$WINS_B" "$TIES" diff --git a/macOS/benchmarks/lib.sh b/macOS/benchmarks/lib.sh new file mode 100755 index 0000000..c59ad70 --- /dev/null +++ b/macOS/benchmarks/lib.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Shared helpers for benchmark scripts. Sourced, not executed directly. +# shellcheck disable=SC2034 # constants consumed by sourcing scripts +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +RESULTS_DIR="$BENCH_DIR/results" +SUITE_VERSION="1.0.0" + +info() { printf ' %s\n' "$*"; } +ok() { printf '\033[32m✓\033[0m %s\n' "$*"; } +warn() { printf '\033[33m!\033[0m %s\n' "$*" >&2; } +die() { printf 'error: %s\n' "$*" >&2; exit 1; } +header() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } + +check_dep() { command -v "$1" >/dev/null 2>&1; } +check_dep_required() { check_dep "$1" || die "required: $1 — install with: brew install $1"; } + +ts_iso() { date -u +"%Y-%m-%dT%H:%M:%SZ"; } +ts_file() { date +"%Y%m%d_%H%M%S"; } + +# Stock macOS `openssl` is LibreSSL, which lacks `speed -seconds`; prefer the +# keg-only brew openssl@3 and accept a PATH openssl only if it is real OpenSSL. +resolve_openssl() { + local cand + for cand in "$(brew --prefix openssl@3 2>/dev/null || true)/bin/openssl" openssl; do + command -v "$cand" >/dev/null 2>&1 || continue + [[ "$("$cand" version 2>/dev/null)" == OpenSSL* ]] && { printf '%s' "$cand"; return 0; } + done + die "no OpenSSL with 'speed -seconds' support found (stock macOS ships LibreSSL) — brew install openssl@3" +} + +ensure_results_dir() { + mkdir -p "$RESULTS_DIR" + # A sudo run (stress-test.sh, for powermetrics) must not leave results/ + # root-owned — later non-sudo suites would finish their whole run and then + # die writing the result file + [[ -n "${SUDO_USER:-}" ]] && chown "$SUDO_USER" "$RESULTS_DIR" + return 0 +} + +# bench_init — shared suite prologue: creates results/, captures +# SYSINFO, sets HOSTNAME_SHORT / STAMP / OUTFILE, prints the system banner. +bench_init() { + local suite="$1" + ensure_results_dir + SYSINFO=$("$BENCH_DIR/collect-sysinfo.sh") + HOSTNAME_SHORT=$(printf '%s' "$SYSINFO" | jq -r '.hostname') + STAMP=$(ts_file) + OUTFILE="$RESULTS_DIR/${suite}_${HOSTNAME_SHORT}_${STAMP}.json" + printf '\n' + ok "System: $(printf '%s' "$SYSINFO" | jq -r '.chip') | $(printf '%s' "$SYSINFO" | jq -r '.memory_gb')GB | macOS $(printf '%s' "$SYSINFO" | jq -r '.macos_version')" + info "Results will be written to: $OUTFILE" +} + +# openssl_sha256_kbs [ncpu] — sha256 16k-block throughput in KB/s +# via `openssl speed`; prints nothing if the output is unparseable. Requires +# OPENSSL_BIN (resolve_openssl). No-arrays branch: bash 3.2 + set -u errors +# on expanding an empty array. +openssl_sha256_kbs() { + local secs="$1" ncpu="${2:-}" raw + if [[ -n "$ncpu" ]]; then + raw=$("$OPENSSL_BIN" speed -elapsed -seconds "$secs" -multi "$ncpu" sha256 2>&1 || true) + else + raw=$("$OPENSSL_BIN" speed -elapsed -seconds "$secs" sha256 2>&1 || true) + fi + printf '%s\n' "$raw" | awk '/^sha256/{gsub(/k$/,"",$7); printf "%.0f", $7+0}' +} diff --git a/macOS/benchmarks/llm-bench.sh b/macOS/benchmarks/llm-bench.sh new file mode 100755 index 0000000..650af57 --- /dev/null +++ b/macOS/benchmarks/llm-bench.sh @@ -0,0 +1,207 @@ +#!/usr/bin/env bash +# Local LLM benchmark — runs the same model through Apple MLX and llama.cpp +# and records prompt-processing (prefill) and generation (decode) tokens/sec. +# Lets you compare two runtimes on one machine, and one runtime across machines. +# +# Usage: bash llm-bench.sh [--quick] [--mlx-only|--llama-only] +# +# Config (override via environment): +# MLX_MODEL HF repo for the MLX runtime (default: gemma-4-12B-it-8bit) +# GGUF_REPO HF repo for the llama.cpp GGUF (default: ggml-org gemma 12b) +# GGUF_QUANT GGUF quant tag (default: Q8_0) +# N_PROMPT prompt tokens for prefill test (default: 512) +# N_GEN tokens to generate (default: 128) +# REPS repetitions to average (default: 3) +# +# Note: MLX 8-bit and GGUF Q8_0 are different quantization schemes — close in +# size/quality but not bit-identical. The tokens/sec rates remain comparable. +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +MLX_MODEL="${MLX_MODEL:-mlx-community/gemma-4-12B-it-8bit}" +GGUF_REPO="${GGUF_REPO:-ggml-org/gemma-4-12b-it-GGUF}" +GGUF_QUANT="${GGUF_QUANT:-Q8_0}" +N_PROMPT="${N_PROMPT:-512}" +N_GEN="${N_GEN:-128}" +REPS="${REPS:-3}" + +QUICK=false +RUN_MLX=true +RUN_LLAMA=true +for arg in "$@"; do + case "$arg" in + --quick) QUICK=true ;; + --mlx-only) RUN_LLAMA=false ;; + --llama-only) RUN_MLX=false ;; + *) die "unknown flag: $arg" ;; + esac +done + +if $QUICK; then + N_GEN=64 + REPS=1 +fi + +check_dep_required jq +check_dep_required curl + +HAS_MLX=false; check_dep mlx_lm.generate && HAS_MLX=true +HAS_LLAMA=false; check_dep llama-bench && HAS_LLAMA=true + +$RUN_MLX && ! $HAS_MLX && { warn "mlx_lm.generate not found (brew install mlx-lm) — skipping MLX"; RUN_MLX=false; } +$RUN_LLAMA && ! $HAS_LLAMA && { warn "llama-bench not found (brew install llama.cpp) — skipping llama.cpp"; RUN_LLAMA=false; } + +$RUN_MLX || $RUN_LLAMA || die "neither MLX nor llama.cpp available — nothing to benchmark" + +bench_init llm +ok "Config: prompt=${N_PROMPT} gen=${N_GEN} reps=${REPS}" +$RUN_MLX && info "MLX model: $MLX_MODEL" +$RUN_LLAMA && info "llama.cpp model: ${GGUF_REPO}:${GGUF_QUANT}" +warn "First run downloads model weights (~13GB per runtime) — this can take a while" + +START_S=$SECONDS + +# Build a sizeable prompt for the prefill measurement (~N_PROMPT tokens). +build_prompt() { + local n="$1" out="" i sentence + sentence="The quick brown fox jumps over the lazy dog while the engineer benchmarks the system. " + # ~14 tokens per sentence; repeat to roughly reach n tokens + local reps=$(( n / 12 + 1 )) + for (( i = 0; i < reps; i++ )); do out+="$sentence"; done + printf '%s' "$out" +} +PROMPT=$(build_prompt "$N_PROMPT") + +# --------------------------------------------------------------------------- +# MLX +# --------------------------------------------------------------------------- +MLX_JSON="null" +if $RUN_MLX; then + header "MLX runtime ($MLX_MODEL)" + MLX_SAMPLES="[]" + MLX_OK=true + for (( r = 1; r <= REPS; r++ )); do + info "Run $r/$REPS..." + OUT=$(mlx_lm.generate \ + --model "$MLX_MODEL" \ + --prompt "$PROMPT" \ + --max-tokens "$N_GEN" \ + 2>&1) || { warn "mlx_lm.generate failed — check model id / network"; MLX_OK=false; break; } + + # || true: a non-matching grep must record null below, not kill the run + # via pipefail (mlx-lm stats format drifts between versions) + PP=$(printf '%s\n' "$OUT" | grep -E '^Prompt:' | grep -oE '[0-9.]+ tokens-per-sec' | grep -oE '[0-9.]+' | head -1 || true) + TG=$(printf '%s\n' "$OUT" | grep -E '^Generation:' | grep -oE '[0-9.]+ tokens-per-sec' | grep -oE '[0-9.]+' | head -1 || true) + MEM=$(printf '%s\n' "$OUT" | grep -E 'Peak memory:' | grep -oE '[0-9.]+ GB' | grep -oE '[0-9.]+' | head -1 || true) + + [[ -z "$PP" ]] && PP="null" + [[ -z "$TG" ]] && TG="null" + [[ -z "$MEM" ]] && MEM="null" + info " prompt=${PP} tok/s generation=${TG} tok/s peak=${MEM}GB" + + MLX_SAMPLES=$(jq \ + --argjson pp "$PP" --argjson tg "$TG" --argjson mem "$MEM" \ + '. + [{pp_tps: $pp, tg_tps: $tg, peak_gb: $mem}]' <<< "$MLX_SAMPLES") + done + + if $MLX_OK; then + MLX_JSON=$(jq -n \ + --arg model "$MLX_MODEL" \ + --argjson samples "$MLX_SAMPLES" \ + '{ + runtime: "mlx", + model: $model, + pp_tps_avg: ([$samples[].pp_tps | select(. != null)] | if length>0 then add/length else null end), + tg_tps_avg: ([$samples[].tg_tps | select(. != null)] | if length>0 then add/length else null end), + peak_gb_max: ([$samples[].peak_gb | select(. != null)] | if length>0 then max else null end), + samples: $samples + }') + ok "MLX avg: prompt=$(jq -r '.pp_tps_avg // "n/a"' <<< "$MLX_JSON") tok/s generation=$(jq -r '.tg_tps_avg // "n/a"' <<< "$MLX_JSON") tok/s" + fi +fi + +# --------------------------------------------------------------------------- +# llama.cpp +# --------------------------------------------------------------------------- +LLAMA_JSON="null" +if $RUN_LLAMA; then + header "llama.cpp runtime (${GGUF_REPO}:${GGUF_QUANT})" + + # llama-bench only takes local model paths (--hf-repo belongs to + # llama-cli/llama-server), so resolve the quant's .gguf filename via the HF + # API and download it once to a local cache. + GGUF_PATH="" + GGUF_FILE=$(curl -fsS "https://huggingface.co/api/models/${GGUF_REPO}" 2>/dev/null \ + | jq -r --arg q "$GGUF_QUANT" \ + '[.siblings[].rfilename | select(test("(?i)" + $q + "\\.gguf$"))] | first // empty' \ + || true) + if [[ -z "$GGUF_FILE" ]]; then + warn "no single-file ${GGUF_QUANT}.gguf found in ${GGUF_REPO} — skipping llama.cpp" + warn "Override with: GGUF_REPO= GGUF_QUANT= bash llm-bench.sh" + else + GGUF_CACHE="${GGUF_CACHE:-$HOME/.cache/llama.cpp}" + mkdir -p "$GGUF_CACHE" + GGUF_PATH="$GGUF_CACHE/${GGUF_FILE##*/}" + if [[ ! -f "$GGUF_PATH" ]]; then + info "Downloading ${GGUF_REPO}/${GGUF_FILE} to ${GGUF_CACHE}..." + curl -fL --progress-bar -o "$GGUF_PATH" \ + "https://huggingface.co/${GGUF_REPO}/resolve/main/${GGUF_FILE}" \ + || { rm -f "$GGUF_PATH"; GGUF_PATH=""; warn "GGUF download failed — skipping llama.cpp"; } + fi + fi + + if [[ -n "$GGUF_PATH" ]]; then + info "Running llama-bench (-p ${N_PROMPT} -n ${N_GEN} -r ${REPS})..." + LB_ERR=$(mktemp) + if LB_OUT=$(llama-bench -m "$GGUF_PATH" \ + -p "$N_PROMPT" -n "$N_GEN" -r "$REPS" \ + -o json 2>"$LB_ERR") && [[ -n "$LB_OUT" ]]; then + + # pp test has n_prompt>0,n_gen==0; tg test has n_gen>0,n_prompt==0 + PP_TPS=$(jq '[.[] | select((.n_prompt|tonumber) > 0 and (.n_gen|tonumber) == 0) | .avg_ts | tonumber?] | if length>0 then add/length else null end' <<< "$LB_OUT") + TG_TPS=$(jq '[.[] | select((.n_gen|tonumber) > 0 and (.n_prompt|tonumber) == 0) | .avg_ts | tonumber?] | if length>0 then add/length else null end' <<< "$LB_OUT") + + LLAMA_JSON=$(jq -n \ + --arg repo "${GGUF_REPO}:${GGUF_QUANT}" \ + --argjson pp "${PP_TPS:-null}" \ + --argjson tg "${TG_TPS:-null}" \ + '{ runtime: "llama.cpp", model: $repo, pp_tps_avg: $pp, tg_tps_avg: $tg }') + ok "llama.cpp avg: prompt=$(jq -r '.pp_tps_avg // "n/a"' <<< "$LLAMA_JSON") tok/s generation=$(jq -r '.tg_tps_avg // "n/a"' <<< "$LLAMA_JSON") tok/s" + else + warn "llama-bench failed — last stderr lines:" + tail -5 "$LB_ERR" >&2 || true + fi + rm -f "$LB_ERR" + fi +fi + +# --------------------------------------------------------------------------- +# Write results +# --------------------------------------------------------------------------- +header "Writing results" +DURATION_S=$(( SECONDS - START_S )) + +jq -n \ + --argjson sysinfo "$SYSINFO" \ + --argjson mlx "$MLX_JSON" \ + --argjson llama "$LLAMA_JSON" \ + --argjson nprompt "$N_PROMPT" \ + --argjson ngen "$N_GEN" \ + --argjson reps "$REPS" \ + --arg ts "$(ts_iso)" \ + --argjson dur "$DURATION_S" \ + --arg sv "$SUITE_VERSION" \ + '{ + metadata: { suite: "llm", timestamp: $ts, duration_s: $dur, suite_version: $sv }, + sysinfo: $sysinfo, + config: { n_prompt: $nprompt, n_gen: $ngen, reps: $reps }, + mlx: $mlx, + llama_cpp: $llama + }' > "$OUTFILE" + +ok "Done in ${DURATION_S}s" +ok "Results: $OUTFILE" +printf '\n' diff --git a/macOS/benchmarks/omlx-bench.sh b/macOS/benchmarks/omlx-bench.sh new file mode 100755 index 0000000..dcee22d --- /dev/null +++ b/macOS/benchmarks/omlx-bench.sh @@ -0,0 +1,268 @@ +#!/usr/bin/env bash +# oMLX concurrency benchmark — measures continuous-batching throughput. +# +# Starts a local oMLX inference server, sweeps concurrency levels by firing N +# parallel completion requests at its OpenAI-compatible API, and records +# aggregate tokens/sec at each level. Single-stream MLX/llama.cpp can't show +# this — aggregate throughput should climb with concurrency until the batch +# scheduler saturates, which is oMLX's whole point. +# +# Usage: bash omlx-bench.sh [--quick] +# +# Config (override via environment): +# OMLX_MODEL_DIR directory of MLX models to serve (default: ~/models) +# OMLX_MODEL model id to request; auto-discovered from /v1/models if unset +# OMLX_PORT server port (default: 8000) +# CONCURRENCY space-separated levels to sweep (default: "1 4 8 16") +# N_GEN tokens to generate per request (default: 128) +# READY_TIMEOUT seconds to wait for model load (default: 180) +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +OMLX_MODEL_DIR="${OMLX_MODEL_DIR:-$HOME/models}" +OMLX_MODEL="${OMLX_MODEL:-}" +OMLX_PORT="${OMLX_PORT:-8000}" +CONCURRENCY="${CONCURRENCY:-1 4 8 16}" +N_GEN="${N_GEN:-128}" +READY_TIMEOUT="${READY_TIMEOUT:-180}" + +QUICK=false +[[ "${1:-}" == "--quick" ]] && QUICK=true +if $QUICK; then + CONCURRENCY="1 4" + N_GEN=64 +fi + +check_dep_required jq +check_dep_required curl + +# Resolve the omlx CLI: PATH first, then the menu-bar app's shim. May be empty +# if only the app is installed and already serving — we reuse that server then. +OMLX_BIN="" +if command -v omlx >/dev/null 2>&1; then + OMLX_BIN="omlx" +elif [[ -x "$HOME/.omlx/bin/omlx" ]]; then + OMLX_BIN="$HOME/.omlx/bin/omlx" +fi + +BASE_URL="http://localhost:${OMLX_PORT}" +# max_concurrent_requests must cover the highest sweep level +MAXCONC=$(printf '%s\n' $CONCURRENCY | sort -n | tail -1) + +bench_init omlx + +WORKDIR=$(mktemp -d) +SERVER_PID="" +SERVER_LOG="$WORKDIR/server.log" + +STARTED_SERVER=false +cleanup() { + # Only tear down a server we started ourselves; leave the app's server alone. + if $STARTED_SERVER && [[ -n "$SERVER_PID" ]]; then + info "Stopping oMLX server (pid $SERVER_PID)..." + kill "$SERVER_PID" 2>/dev/null || true + wait "$SERVER_PID" 2>/dev/null || true + fi + rm -rf "$WORKDIR" +} +trap cleanup EXIT +# exit so the sweep can't resume against the deleted WORKDIR after Ctrl-C; +# cleanup runs exactly once via the EXIT trap +trap 'exit 130' INT TERM + +# perl gives sub-second wall-clock timing portably (bash 3.2 has no EPOCHREALTIME) +now_s() { + if command -v perl >/dev/null 2>&1; then + perl -MTime::HiRes=time -e 'printf "%.4f\n", time' + else + date +%s + fi +} + +ok "Sweep: concurrency [${CONCURRENCY}] gen=${N_GEN} tok/req max_concurrent=${MAXCONC}" + +# --------------------------------------------------------------------------- +# Reuse a running server (e.g. the menu-bar app) or start our own +# --------------------------------------------------------------------------- +if curl -fsS "${BASE_URL}/v1/models" >/dev/null 2>&1; then + header "Reusing oMLX server on ${BASE_URL}" + info "A server is already running (likely the menu-bar app) — not starting another" + warn "Its own --max-concurrent-requests caps batching; sweep levels above it just queue" +else + [[ -n "$OMLX_BIN" ]] || die "no oMLX server on ${BASE_URL} and no omlx CLI found — launch the menu-bar app or install omlx" + [[ -d "$OMLX_MODEL_DIR" ]] || die "model dir not found: $OMLX_MODEL_DIR — place MLX models under /// or set OMLX_MODEL_DIR" + header "Starting oMLX server" + "$OMLX_BIN" serve \ + --model-dir "$OMLX_MODEL_DIR" \ + --port "$OMLX_PORT" \ + --max-concurrent-requests "$MAXCONC" \ + >"$SERVER_LOG" 2>&1 & + SERVER_PID=$! + STARTED_SERVER=true + info "Server pid: $SERVER_PID (log: $SERVER_LOG)" +fi + +# --------------------------------------------------------------------------- +# Wait for readiness — poll /v1/models until a model is listed +# --------------------------------------------------------------------------- +info "Waiting for model load (timeout ${READY_TIMEOUT}s)..." +DISCOVERED="" +READY=false +for (( t = 0; t < READY_TIMEOUT; t += 3 )); do + if $STARTED_SERVER && ! kill -0 "$SERVER_PID" 2>/dev/null; then + warn "Server exited early — last log lines:" + tail -20 "$SERVER_LOG" >&2 || true + die "oMLX server failed to start" + fi + MODELS=$(curl -fsS "${BASE_URL}/v1/models" 2>/dev/null || true) + if [[ -n "$MODELS" ]]; then + DISCOVERED=$(printf '%s' "$MODELS" | jq -r '.data[0].id // empty' 2>/dev/null || true) + # With an explicit OMLX_MODEL a responding API is enough — a lazy-loading + # server lists nothing until the first request + if [[ -n "$DISCOVERED" || -n "$OMLX_MODEL" ]]; then READY=true; break; fi + fi + sleep 3 +done +if ! $READY; then + if $STARTED_SERVER; then + tail -20 "$SERVER_LOG" >&2 || true + die "no model available after ${READY_TIMEOUT}s — check $SERVER_LOG and that MLX models live under $OMLX_MODEL_DIR" + else + die "reused server on ${BASE_URL} lists no models — load a model in the oMLX app first, or set OMLX_MODEL" + fi +fi + +MODEL="${OMLX_MODEL:-$DISCOVERED}" +ok "Serving model: $MODEL" + +# --------------------------------------------------------------------------- +# Request body (shared across all requests) +# --------------------------------------------------------------------------- +BODY="$WORKDIR/body.json" +jq -n --arg model "$MODEL" --argjson n "$N_GEN" \ + '{ + model: $model, + messages: [ + {role: "user", content: "Write a detailed technical explanation of how continuous batching improves LLM inference throughput on Apple Silicon. Be thorough."} + ], + max_tokens: $n, + temperature: 0.0, + stream: false + }' > "$BODY" + +fire_one() { + # $1 = output file for this request's JSON response; returns curl's status + # so callers can count failures instead of silently scoring them as 0 tokens + if ! curl -fsS -X POST "${BASE_URL}/v1/chat/completions" \ + -H "Content-Type: application/json" \ + --data @"$BODY" \ + -o "$1" 2>/dev/null; then + printf '{"error":"request failed"}' > "$1" + return 1 + fi +} + +# Warmup (loads weights into the hot cache, primes the scheduler) +header "Warmup" +fire_one "$WORKDIR/warmup.json" || true +WU_TOKENS=$(jq '.usage.completion_tokens // 0' "$WORKDIR/warmup.json") +[[ "$WU_TOKENS" -gt 0 ]] || warn "warmup returned no tokens — check $WORKDIR/warmup.json" +ok "Warmup complete (${WU_TOKENS} tokens)" + +# --------------------------------------------------------------------------- +# Concurrency sweep +# --------------------------------------------------------------------------- +header "Concurrency sweep" +printf ' %-12s %-14s %-18s %-14s\n' "Concurrency" "Wall (s)" "Total tokens" "Aggregate tok/s" +printf ' %s\n' "------------------------------------------------------------" + +SWEEP_JSON="[]" +for C in $CONCURRENCY; do + PIDS=() + T0=$(now_s) + for (( i = 0; i < C; i++ )); do + fire_one "$WORKDIR/resp_${C}_${i}.json" & + PIDS+=("$!") + done + FAILED=0 + for p in "${PIDS[@]}"; do wait "$p" || FAILED=$(( FAILED + 1 )); done + T1=$(now_s) + + WALL=$(jq -n "$T1 - $T0") + TOTAL_TOK=0 + for (( i = 0; i < C; i++ )); do + tk=$(jq '.usage.completion_tokens // 0' "$WORKDIR/resp_${C}_${i}.json") + TOTAL_TOK=$(( TOTAL_TOK + tk )) + done + AGG_TPS=$(jq -n "if $WALL > 0 then $TOTAL_TOK / $WALL else 0 end") + + printf ' %-12s %-14s %-18s %-14s\n' \ + "$C" "$(printf '%.2f' "$WALL")" "$TOTAL_TOK" "$(printf '%.1f' "$AGG_TPS")" + (( FAILED > 0 )) && warn "concurrency ${C}: ${FAILED}/${C} requests failed — level excluded from peak/speedup" + + SWEEP_JSON=$(jq \ + --argjson c "$C" --argjson wall "$WALL" \ + --argjson tok "$TOTAL_TOK" --argjson tps "$AGG_TPS" \ + --argjson failed "$FAILED" \ + '. + [{concurrency: $c, wall_s: $wall, total_completion_tokens: $tok, aggregate_tps: $tps, failed_requests: $failed}]' \ + <<< "$SWEEP_JSON") +done + +# --------------------------------------------------------------------------- +# Summary +# --------------------------------------------------------------------------- +# Levels with failed requests understate throughput — keep them out of the +# headline numbers +SINGLE_TPS=$(jq '[.[] | select(.concurrency == 1 and .failed_requests == 0) | .aggregate_tps] | first // null' <<< "$SWEEP_JSON") +PEAK=$(jq '[.[] | select(.failed_requests == 0)] | if length > 0 then max_by(.aggregate_tps) else null end' <<< "$SWEEP_JSON") +PEAK_TPS=$(jq '.aggregate_tps // null' <<< "$PEAK") +PEAK_CONC=$(jq '.concurrency // null' <<< "$PEAK") +SPEEDUP="null" +if [[ "$SINGLE_TPS" != "null" && "$PEAK_TPS" != "null" ]]; then + SPEEDUP=$(jq -n "if $SINGLE_TPS > 0 then $PEAK_TPS / $SINGLE_TPS else null end") +fi + +header "Summary" +if [[ "$SINGLE_TPS" == "null" ]]; then + ok "Single-stream: n/a (no clean concurrency=1 sample)" +else + ok "Single-stream: $(printf '%.1f' "$SINGLE_TPS") tok/s" +fi +if [[ "$PEAK_TPS" == "null" ]]; then + warn "every sweep level had failed requests — peak/speedup not recorded" +else + ok "Peak aggregate: $(printf '%.1f' "$PEAK_TPS") tok/s @ concurrency ${PEAK_CONC}" +fi +[[ "$SPEEDUP" != "null" ]] && ok "Batching speedup: $(printf '%.2fx' "$SPEEDUP")" + +# --------------------------------------------------------------------------- +# Write results +# --------------------------------------------------------------------------- +jq -n \ + --argjson sysinfo "$SYSINFO" \ + --arg model "$MODEL" \ + --argjson maxc "$MAXCONC" \ + --argjson ngen "$N_GEN" \ + --argjson single "${SINGLE_TPS:-null}" \ + --argjson peak "$PEAK_TPS" \ + --argjson peakc "$PEAK_CONC" \ + --argjson speedup "${SPEEDUP:-null}" \ + --argjson sweep "$SWEEP_JSON" \ + --arg ts "$(ts_iso)" \ + --arg sv "$SUITE_VERSION" \ + '{ + metadata: { suite: "omlx", timestamp: $ts, suite_version: $sv }, + sysinfo: $sysinfo, + config: { model: $model, max_concurrent_requests: $maxc, n_gen: $ngen }, + single_stream_tps: $single, + peak_aggregate_tps: $peak, + peak_concurrency: $peakc, + batching_speedup: $speedup, + sweep: $sweep + }' > "$OUTFILE" + +ok "Results: $OUTFILE" +printf '\n' diff --git a/macOS/benchmarks/standardized.sh b/macOS/benchmarks/standardized.sh new file mode 100755 index 0000000..25620bf --- /dev/null +++ b/macOS/benchmarks/standardized.sh @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# Standardized benchmark wrapper — runs Geekbench 6, Cinebench, and Blender +# Benchmark CLIs (whichever are installed) and records their scores. +# +# Each benchmark is best-effort: a missing app or an unparseable result is +# recorded as null with a note, never aborts the run. CLI output formats vary +# by version, so scores are captured from stdout and also saved raw for audit. +# +# Usage: bash standardized.sh [--cpu-only] +# --cpu-only skip GPU/compute sub-tests where the CLI allows it +# +# Geekbench note: the free tier uploads results to the public Geekbench Browser +# and returns a URL; a Pro license enables offline --export-json. Both handled. +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +CPU_ONLY=false +[[ "${1:-}" == "--cpu-only" ]] && CPU_ONLY=true + +check_dep_required jq + +bench_init standardized +RAWDIR="$RESULTS_DIR/standardized_${HOSTNAME_SHORT}_${STAMP}_raw" +info "Raw CLI output saved to: $RAWDIR" +mkdir -p "$RAWDIR" + +START_S=$SECONDS + +# --------------------------------------------------------------------------- +# Geekbench 6 +# --------------------------------------------------------------------------- +GEEKBENCH_JSON="null" +GB_BIN=$(find "/Applications/Geekbench 6.app/Contents/Resources" -maxdepth 1 -name 'geekbench6' -type f 2>/dev/null | head -1 || true) + +if [[ -n "$GB_BIN" ]]; then + header "Geekbench 6" + GB_JSON_OUT="$RAWDIR/geekbench6.json" + GB_RAW="$RAWDIR/geekbench6.txt" + + # Try Pro offline JSON export first; fall back to default (upload + URL) + if "$GB_BIN" --cpu --export-json "$GB_JSON_OUT" >"$GB_RAW" 2>&1 && [[ -s "$GB_JSON_OUT" ]]; then + GB_SINGLE=$(jq '.sections // [] | map(select(.name=="Single-Core")) | .[0].score // null' "$GB_JSON_OUT" 2>/dev/null || printf 'null') + GB_MULTI=$(jq '.sections // [] | map(select(.name=="Multi-Core")) | .[0].score // null' "$GB_JSON_OUT" 2>/dev/null || printf 'null') + GEEKBENCH_JSON=$(jq -n --argjson s "${GB_SINGLE:-null}" --argjson m "${GB_MULTI:-null}" \ + '{ single_core: $s, multi_core: $m, mode: "pro-json" }') + ok "Geekbench CPU: single=${GB_SINGLE} multi=${GB_MULTI}" + else + # Free tier: the first invocation may have completed the full run and + # printed the browser URL before the Pro-only export failed — reuse it + # instead of paying a second multi-minute benchmark (and append on the + # re-run so the first run's output survives for audit) + GB_URL=$(grep -oE 'https://browser\.geekbench\.com/[^ ]+' "$GB_RAW" | head -1 || true) + if [[ -z "$GB_URL" ]]; then + "$GB_BIN" --cpu >>"$GB_RAW" 2>&1 || true + GB_URL=$(grep -oE 'https://browser\.geekbench\.com/[^ ]+' "$GB_RAW" | head -1 || true) + fi + if [[ -n "$GB_URL" ]]; then + GEEKBENCH_JSON=$(jq -n --arg url "$GB_URL" '{ result_url: $url, mode: "free-upload", note: "scores in the Geekbench Browser" }') + ok "Geekbench result: $GB_URL" + else + warn "Geekbench produced no parseable result — see $GB_RAW" + GEEKBENCH_JSON=$(jq -n '{ error: "no score/url parsed" }') + fi + fi +else + info "Geekbench 6 not installed (brew install --cask geekbench) — skipping" +fi + +# --------------------------------------------------------------------------- +# Geekbench AI (banff CLI) +# --------------------------------------------------------------------------- +GEEKBENCH_AI_JSON="null" +GBAI_BIN="" +for cand in banff_aarch64 banff; do + p="/Applications/Geekbench AI.app/Contents/Resources/$cand" + [[ -x "$p" ]] && { GBAI_BIN="$p"; break; } +done + +if [[ -n "$GBAI_BIN" ]]; then + header "Geekbench AI" + GBAI_RAW="$RAWDIR/geekbench_ai.txt" + # Free CLI runs the inference benchmark and uploads; scores land in the + # Geekbench AI Browser and the result URL is printed to stdout. + "$GBAI_BIN" --ai >"$GBAI_RAW" 2>&1 || true + GBAI_URL=$(grep -oE 'https://browser\.geekbench\.com/ai/[^ ]+' "$GBAI_RAW" | head -1 || true) + if [[ -n "$GBAI_URL" ]]; then + GEEKBENCH_AI_JSON=$(jq -n --arg url "$GBAI_URL" '{ result_url: $url, mode: "free-upload", note: "single/half/quantized scores in the Geekbench AI Browser" }') + ok "Geekbench AI result: $GBAI_URL" + else + warn "Geekbench AI produced no parseable result — see $GBAI_RAW" + GEEKBENCH_AI_JSON=$(jq -n '{ error: "no score/url parsed" }') + fi +else + info "Geekbench AI not installed (brew install --cask geekbench-ai) — skipping" +fi + +# --------------------------------------------------------------------------- +# Cinebench +# --------------------------------------------------------------------------- +CINEBENCH_JSON="null" +# binary sits at depth 4: /Applications/Cinebench.app/Contents/MacOS/Cinebench +CB_BIN=$(find /Applications -maxdepth 4 -name 'Cinebench' -type f -path '*Contents/MacOS*' 2>/dev/null | head -1 || true) + +if [[ -n "$CB_BIN" ]]; then + header "Cinebench" + # Each run parses its own raw file — a shared file would let a failed + # single-core run silently pick up the multi-core score. + CB_MULTI_RAW="$RAWDIR/cinebench-multi.txt" + # CLI runs the multi-threaded CPU test and prints "CB " + "$CB_BIN" g_CinebenchCpuXTest=true >"$CB_MULTI_RAW" 2>&1 || true + CB_MULTI=$(grep -oE 'CB[[:space:]]+[0-9]+' "$CB_MULTI_RAW" | grep -oE '[0-9]+' | tail -1 || true) + [[ -z "$CB_MULTI" ]] && CB_MULTI="null" + + # Both Cinebench passes are CPU tests — --cpu-only does not skip them + CB_SINGLE_RAW="$RAWDIR/cinebench-single.txt" + "$CB_BIN" g_CinebenchCpu1Test=true >"$CB_SINGLE_RAW" 2>&1 || true + CB_SINGLE=$(grep -oE 'CB[[:space:]]+[0-9]+' "$CB_SINGLE_RAW" | grep -oE '[0-9]+' | tail -1 || true) + [[ -z "$CB_SINGLE" ]] && CB_SINGLE="null" + + CINEBENCH_JSON=$(jq -n --argjson s "$CB_SINGLE" --argjson m "$CB_MULTI" \ + '{ cpu_single: $s, cpu_multi: $m, note: "CLI score format varies by Cinebench version — verify against raw output" }') + ok "Cinebench CPU: single=${CB_SINGLE} multi=${CB_MULTI}" +else + info "Cinebench not installed (bash macOS/install-cinebench.sh) — skipping" +fi + +# --------------------------------------------------------------------------- +# Blender Benchmark +# --------------------------------------------------------------------------- +BLENDER_JSON="null" +BB_BIN=$(find "/Applications/Blender Benchmark.app" -name 'benchmark-launcher-cli' -type f 2>/dev/null | head -1 || true) + +if [[ -n "$BB_BIN" ]] && ! $CPU_ONLY; then + header "Blender Benchmark (Metal GPU)" + BB_RAW="$RAWDIR/blender.json" + # Resolve latest available Blender version, then run the standard scenes on Metal + BB_VER=$("$BB_BIN" blender list 2>/dev/null | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | tail -1 || true) + if [[ -n "$BB_VER" ]]; then + # The benchmark subcommand does not auto-download; fetch the runtime and + # scenes explicitly first (no-ops when already cached) + info "Blender $BB_VER — downloading runtime + scenes (first run only)..." + "$BB_BIN" blender download "$BB_VER" >/dev/null 2>&1 || true + "$BB_BIN" scenes download -b "$BB_VER" monster junkshop classroom >/dev/null 2>&1 || true + "$BB_BIN" benchmark monster junkshop classroom \ + --blender-version "$BB_VER" \ + --device-type METAL \ + --json >"$BB_RAW" 2>/dev/null || true + if [[ -s "$BB_RAW" ]]; then + # samples_per_minute per scene; sum for an overall figure + BB_TOTAL=$(jq '[.[].stats.samples_per_minute // empty] | if length>0 then add else null end' "$BB_RAW" 2>/dev/null || printf 'null') + BLENDER_JSON=$(jq -n --argjson t "${BB_TOTAL:-null}" --slurpfile raw "$BB_RAW" \ + '{ total_samples_per_minute: $t, scenes: ($raw[0] | map({scene: .scene.label, samples_per_minute: .stats.samples_per_minute})) }' 2>/dev/null \ + || jq -n --argjson t "${BB_TOTAL:-null}" '{ total_samples_per_minute: $t }') + ok "Blender total: ${BB_TOTAL} samples/min" + else + warn "Blender Benchmark produced no output — see $BB_RAW" + fi + else + warn "Could not resolve a Blender version from the launcher" + fi +elif [[ -z "$BB_BIN" ]]; then + info "Blender Benchmark not installed (brew install --cask blender-benchmark) — skipping" +fi + +# --------------------------------------------------------------------------- +# Write results +# --------------------------------------------------------------------------- +header "Writing results" +DURATION_S=$(( SECONDS - START_S )) + +jq -n \ + --argjson sysinfo "$SYSINFO" \ + --argjson geekbench "$GEEKBENCH_JSON" \ + --argjson geekbenchai "$GEEKBENCH_AI_JSON" \ + --argjson cinebench "$CINEBENCH_JSON" \ + --argjson blender "$BLENDER_JSON" \ + --arg ts "$(ts_iso)" \ + --argjson dur "$DURATION_S" \ + --arg sv "$SUITE_VERSION" \ + '{ + metadata: { suite: "standardized", timestamp: $ts, duration_s: $dur, suite_version: $sv }, + sysinfo: $sysinfo, + geekbench6: $geekbench, + geekbench_ai: $geekbenchai, + cinebench: $cinebench, + blender_benchmark: $blender + }' > "$OUTFILE" + +ok "Done in ${DURATION_S}s" +ok "Results: $OUTFILE" +printf '\n' diff --git a/macOS/benchmarks/stress-test.sh b/macOS/benchmarks/stress-test.sh new file mode 100755 index 0000000..1d2fd6a --- /dev/null +++ b/macOS/benchmarks/stress-test.sh @@ -0,0 +1,201 @@ +#!/usr/bin/env bash +# Sustained CPU stress test with thermal/throttle detection. +# Exercises all cores for DURATION seconds and samples throughput every INTERVAL +# seconds to detect if performance degrades under heat (throttling). +# +# Usage: bash stress-test.sh [DURATION_SECONDS] +# Default duration: 300s (5 minutes) +# Run with sudo for CPU frequency + power data via powermetrics. +set -euo pipefail + +BENCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" +# shellcheck source=lib.sh +source "$BENCH_DIR/lib.sh" + +DURATION=${1:-300} +INTERVAL=30 +THROTTLE_THRESHOLD="0.90" + +check_dep_required jq +check_dep_required stress-ng +# shellcheck disable=SC2034 # consumed by openssl_sha256_kbs in lib.sh +OPENSSL_BIN=$(resolve_openssl) + +bench_init stress +NCPU=$(sysctl -n hw.logicalcpu) + +STRESS_PID="" +SAMPLES_JSON="[]" + +cleanup() { + if [[ -n "$STRESS_PID" ]]; then + kill "$STRESS_PID" 2>/dev/null || true + wait "$STRESS_PID" 2>/dev/null || true + fi +} +trap cleanup EXIT +# A non-exiting INT/TERM handler would kill the load but let the sample loop +# keep running on an idle machine and write a bogus throttled:false result; +# exiting here routes cleanup through the EXIT trap exactly once +trap 'exit 130' INT TERM + +HAS_SUDO_N=false +if sudo -n true 2>/dev/null; then + HAS_SUDO_N=true + info "sudo available — will collect CPU frequency and power via powermetrics" +else + warn "No passwordless sudo — using throughput-ratio proxy for throttle detection" + warn "Run 'sudo bash stress-test.sh' for power/frequency data" +fi + +ok "Duration: ${DURATION}s | Sample interval: ${INTERVAL}s | Cores: ${NCPU}" + +# --------------------------------------------------------------------------- +# Start stress load +# --------------------------------------------------------------------------- +header "Starting stress load on all ${NCPU} cores" +# --cpu 0 = one stressor per logical CPU; sha512 exercises integer + vector +# units. The extra 30s covers the baseline settle/measure time so the last +# sample still lands under load. +stress-ng --cpu 0 --cpu-method sha512 --timeout "$(( DURATION + 30 ))s" & +STRESS_PID=$! +info "Stress PID: $STRESS_PID" + +# --------------------------------------------------------------------------- +# Baseline measurement (under load) +# --------------------------------------------------------------------------- +# The baseline must be taken while the stressors are running: an idle baseline +# reads a single-core-boost P-core, and the probe then contends with NCPU +# stressors on every sample — scheduler contention alone would read as a +# "throttle". Let clocks settle at the sustained all-core level first; +# throttling then shows up as a decline relative to this early-load figure. +header "Baseline throughput (5s, under load)" +sleep 10 +BASELINE_KBS=$(openssl_sha256_kbs 5) +if [[ -z "$BASELINE_KBS" || "$BASELINE_KBS" == 0 ]]; then + die "could not establish a baseline sha256 throughput (openssl parse failed) — cannot compute throttle ratios" +fi +ok "Loaded baseline sha256: ${BASELINE_KBS} KB/s" + +# --------------------------------------------------------------------------- +# Sample loop +# --------------------------------------------------------------------------- +header "Sampling every ${INTERVAL}s for ${DURATION}s" +printf ' %-8s %-20s %-8s %-14s %-14s\n' "Elapsed" "sha256 KB/s" "Ratio" "CPU MHz" "Power W" +printf ' %s\n' "--------------------------------------------------------------" + +SAMPLE_COUNT=$(( DURATION / INTERVAL )) +if (( SAMPLE_COUNT < 1 )); then SAMPLE_COUNT=1; fi +for (( i = 1; i <= SAMPLE_COUNT; i++ )); do + sleep "$INTERVAL" + + ELAPSED=$(( i * INTERVAL )) + + # 1-second throughput measurement (brief window, parallel to stress load) + CUR_KBS=$(openssl_sha256_kbs 1) + [[ -z "$CUR_KBS" ]] && CUR_KBS=0 + + RATIO=$(jq -n "${CUR_KBS} / ${BASELINE_KBS}") + THROTTLE_FLAG="" + if (( $(jq -n "if ${RATIO} < ${THROTTLE_THRESHOLD} then 1 else 0 end") )); then + THROTTLE_FLAG=" !! THROTTLE" + fi + + CPU_FREQ_MHZ="null" + POWER_W="null" + if $HAS_SUDO_N; then + PM_OUT=$(sudo powermetrics -n 1 -i 200 \ + --samplers cpu_power --hide-cpu-duty-cycle 2>/dev/null || true) + # Apple Silicon: "P0-Cluster HW active frequency: 3204 MHz" (or "P-Cluster" + # on base chips) and "CPU Power: 4382 mW" — there is no Intel-style + # "Package power:" line + RAW_FREQ=$(printf '%s\n' "$PM_OUT" \ + | awk '/^P[0-9]*-Cluster HW active frequency:/{print $(NF-1)+0; exit}' \ + || printf '') + [[ -n "$RAW_FREQ" ]] && CPU_FREQ_MHZ="$RAW_FREQ" + RAW_WATTS=$(printf '%s\n' "$PM_OUT" \ + | awk '/^CPU Power:/{printf "%.2f", $3/1000; exit}' \ + || printf '') + [[ -n "$RAW_WATTS" ]] && POWER_W="$RAW_WATTS" + fi + + FREQ_DISPLAY="${CPU_FREQ_MHZ:-n/a}" + WATTS_DISPLAY="${POWER_W:-n/a}" + [[ "$FREQ_DISPLAY" == "null" ]] && FREQ_DISPLAY="n/a" + [[ "$WATTS_DISPLAY" == "null" ]] && WATTS_DISPLAY="n/a" + + printf ' %-8s %-20s %-8s %-14s %-14s%s\n' \ + "${ELAPSED}s" "${CUR_KBS}" "$(printf '%.3f' "$RATIO")" \ + "$FREQ_DISPLAY" "$WATTS_DISPLAY" "$THROTTLE_FLAG" + + SAMPLE=$(jq -n \ + --arg ts "$(ts_iso)" \ + --argjson elapsed "$ELAPSED" \ + --argjson sha_kbs "$CUR_KBS" \ + --argjson base_kbs "$BASELINE_KBS" \ + --argjson ratio "$RATIO" \ + --argjson freq "$CPU_FREQ_MHZ" \ + --argjson watts "$POWER_W" \ + '{ts: $ts, elapsed_s: $elapsed, sha256_kbs: $sha_kbs, + baseline_kbs: $base_kbs, throttle_ratio: $ratio, + cpu_freq_mhz: $freq, power_watts: $watts}') + + SAMPLES_JSON=$(jq --argjson s "$SAMPLE" '. + [$s]' <<< "$SAMPLES_JSON") +done + +# Wait for stress to finish cleanly (it should have already timed out) +wait "$STRESS_PID" 2>/dev/null || true +STRESS_PID="" + +# --------------------------------------------------------------------------- +# Throttle analysis +# --------------------------------------------------------------------------- +header "Throttle analysis" + +MIN_RATIO=$(jq '[.[].throttle_ratio] | min' <<< "$SAMPLES_JSON") +MAX_RATIO=$(jq '[.[].throttle_ratio] | max' <<< "$SAMPLES_JSON") +AVG_RATIO=$(jq '[.[].throttle_ratio] | add / length' <<< "$SAMPLES_JSON") + +THROTTLED="false" +if (( $(jq -n "if ${MIN_RATIO} < ${THROTTLE_THRESHOLD} then 1 else 0 end") )); then + THROTTLED="true" + warn "Throttling detected — min ratio: $(printf '%.3f' "$MIN_RATIO") (threshold: $THROTTLE_THRESHOLD)" +else + ok "No throttling detected — min ratio: $(printf '%.3f' "$MIN_RATIO")" +fi + +ok "Ratio range: $(printf '%.3f' "$MIN_RATIO") – $(printf '%.3f' "$MAX_RATIO") avg: $(printf '%.3f' "$AVG_RATIO")" + +# --------------------------------------------------------------------------- +# Write result file +# --------------------------------------------------------------------------- +jq -n \ + --argjson sysinfo "$SYSINFO" \ + --argjson baseline "$BASELINE_KBS" \ + --argjson ncpu "$NCPU" \ + --argjson duration "$DURATION" \ + --argjson interval "$INTERVAL" \ + --arg throttled "$THROTTLED" \ + --argjson min_ratio "$MIN_RATIO" \ + --argjson avg_ratio "$AVG_RATIO" \ + --arg threshold "$THROTTLE_THRESHOLD" \ + --argjson samples "$SAMPLES_JSON" \ + --arg timestamp "$(ts_iso)" \ + --arg sv "$SUITE_VERSION" \ + '{ + metadata: { suite: "stress", timestamp: $timestamp, suite_version: $sv }, + sysinfo: $sysinfo, + config: { duration_s: $duration, interval_s: $interval, num_cores: $ncpu }, + baseline_sha256_kbs: $baseline, + throttle_detection: { + throttled: ($throttled == "true"), + min_ratio: $min_ratio, + avg_ratio: $avg_ratio, + threshold_ratio: ($threshold | tonumber), + method: "ratio vs early-load baseline (settled clocks, probe contending with stressors)" + }, + samples: $samples + }' > "$OUTFILE" + +ok "Results: $OUTFILE" +printf '\n' diff --git a/macOS/install-cinebench.sh b/macOS/install-cinebench.sh new file mode 100755 index 0000000..a6652d1 --- /dev/null +++ b/macOS/install-cinebench.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Installs Maxon Cinebench from its official .dmg. +# +# Why not the Homebrew cask: Maxon serves a rolling build at a stable URL, so +# the cask's pinned SHA256 routinely goes stale and `brew install --cask +# cinebench` dies with "SHA256 mismatch". This downloads the same DMG directly +# and skips the checksum, which is the only reliable unattended install. +# +# Usage: bash macOS/install-cinebench.sh +# Override the DMG when Maxon ships a new major version: +# CINEBENCH_DMG_URL=https://.../Cinebench20XX_macOS.dmg bash macOS/install-cinebench.sh +set -euo pipefail + +DMG_URL="${CINEBENCH_DMG_URL:-https://mx-app-blob-prod.maxon.net/mx-package-production/website/macos/maxon/cinebench/Cinebench2026_macOS.dmg}" + +source "$(dirname "${BASH_SOURCE[0]:-$0}")/lib-dmg-install.sh" + +install_app_from_dmg Cinebench "$DMG_URL" diff --git a/macOS/install-omlx-app.sh b/macOS/install-omlx-app.sh new file mode 100755 index 0000000..4f71361 --- /dev/null +++ b/macOS/install-omlx-app.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Installs the oMLX macOS menu-bar app from its GitHub Releases .dmg. +# oMLX ships no Homebrew cask — the native app is .dmg-only (it self-updates +# in-app afterward). Do NOT also install the `omlx` brew formula: the app +# bundles its own server and they collide on port 8000 and ~/.omlx/. +# +# Usage: bash macOS/install-omlx-app.sh +set -euo pipefail + +REPO="jundot/omlx" +API="https://api.github.com/repos/${REPO}/releases/latest" + +source "$(dirname "${BASH_SOURCE[0]:-$0}")/lib-dmg-install.sh" + +# Guard before install_app_from_dmg's own check: skip the GitHub API call +# (rate-limited) on idempotent re-runs +if app_installed oMLX; then + info "oMLX.app already in /Applications — it self-updates in-app; skipping download" + exit 0 +fi + +info "Resolving latest oMLX release..." +dmg_urls=$(curl -fsSL "$API" \ + | grep -oE '"browser_download_url":[[:space:]]*"[^"]+\.dmg"' \ + | grep -oE 'https://[^"]+\.dmg' || true) +[[ -n "$dmg_urls" ]] || die "could not find a .dmg asset in the latest release of $REPO" + +# oMLX ships one DMG per macOS generation (e.g. macos15-sequoia, macos26-27), +# with no architecture variants. Each filename embeds a macosNN token; pick the +# build whose NN is the highest that does not exceed this Mac's major version +# (the newest build that still targets ≤ this OS). Fall back to the first asset +# if none carry a parseable version. +os_major=$(sw_vers -productVersion | cut -d. -f1) +best_ver=-1 +dmg_url="" +while IFS= read -r url; do + [[ -n "$url" ]] || continue + v=$(grep -oE 'macos[0-9]+' <<< "$url" | grep -oE '[0-9]+' | sort -n | tail -1 || true) + [[ -n "$v" ]] || continue + if (( v <= os_major && v > best_ver )); then + best_ver="$v" + dmg_url="$url" + fi +done <<< "$dmg_urls" +[[ -n "$dmg_url" ]] || dmg_url=$(head -1 <<< "$dmg_urls") +info "Selected DMG for macOS $os_major: $(basename "$dmg_url")" + +install_app_from_dmg oMLX "$dmg_url" diff --git a/macOS/lib-dmg-install.sh b/macOS/lib-dmg-install.sh new file mode 100755 index 0000000..11d6873 --- /dev/null +++ b/macOS/lib-dmg-install.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# Shared helper for installing a macOS app from a .dmg. Sourced (not executed) +# by install-omlx-app.sh and install-cinebench.sh. Each caller resolves the .dmg +# URL its own way (GitHub Releases API + OS-version selection vs a fixed vendor +# URL); this owns the common guard → deps → download → mount → copy → cleanup. + +info() { printf ' %s\n' "$*"; } +die() { printf 'error: %s\n' "$*" >&2; exit 1; } + +# app_installed — the .app bundle is already in /Applications +app_installed() { [[ -d "/Applications/$1.app" ]]; } + +# install_app_from_dmg +# Skips if /Applications/.app exists. Downloads the DMG, mounts it, +# copies the first *.app at the image root into /Applications, prints the +# success line, and cleans up on exit. Dies on any failure. +install_app_from_dmg() { + local app_name="$1" dmg_url="$2" + local workdir mnt dmg app attach_out actual_mnt + + if app_installed "$app_name"; then + info "${app_name}.app already in /Applications — skipping download" + return 0 + fi + command -v curl >/dev/null || die "curl is required" + command -v hdiutil >/dev/null || die "hdiutil is required (macOS only)" + + workdir=$(mktemp -d) + mnt="$workdir/mnt" + dmg="$workdir/image.dmg" + # Bake the paths into the trap now: it fires at *script* exit, after this + # function has returned and its locals are out of scope — a bare "$mnt" + # reference would then be an unbound-variable error under set -u. + # shellcheck disable=SC2064 # intentional: expand $mnt/$workdir at register time + trap "hdiutil detach '$mnt' >/dev/null 2>&1 || true; rm -rf '$workdir'" EXIT + + info "Downloading $(basename "$dmg_url")..." + curl -fsSL "$dmg_url" -o "$dmg" || die "download failed: $dmg_url" + + info "Mounting disk image..." + mkdir -p "$mnt" + attach_out=$(hdiutil attach "$dmg" -nobrowse -noverify -mountpoint "$mnt") \ + || die "failed to mount $dmg" + # hdiutil silently reuses an existing mount (ignoring -mountpoint) when the + # image is already attached, e.g. previously opened in Finder — trust the + # mount point it reports (tab-separated last column) over the one requested + actual_mnt=$(printf '%s\n' "$attach_out" | awk -F'\t' '$NF ~ /^\// {mp=$NF} END {print mp}') + [[ -n "$actual_mnt" && -d "$actual_mnt" ]] && mnt="$actual_mnt" + # re-register so cleanup detaches the mount actually used + # shellcheck disable=SC2064 + trap "hdiutil detach '$mnt' >/dev/null 2>&1 || true; rm -rf '$workdir'" EXIT + + app=$(find "$mnt" -maxdepth 1 -name '*.app' -print -quit 2>/dev/null || true) + [[ -n "$app" ]] || die "no .app found inside the disk image" + + info "Installing $(basename "$app") to /Applications..." + if ! cp -R "$app" /Applications/ 2>/dev/null; then + die "could not copy to /Applications — drag $(basename "$app") there manually from $dmg_url" + fi + + printf '\033[32m✓\033[0m %s installed. Launch it from /Applications (Gatekeeper may prompt on first run).\n' "$app_name" +} diff --git a/packages.json b/packages.json index 48e7d5c..49e7720 100644 --- a/packages.json +++ b/packages.json @@ -583,6 +583,16 @@ "optional": false, "description": "Track AI coding token usage by task, tool, model, and project" }, + { + "name": "codeburn-menubar", + "tags": ["development", "ai-coding"], + "package_manager": { "macos": "custom" }, + "handled_by_setup": false, + "install_command": "codeburn menubar", + "priority": "none", + "optional": false, + "description": "codeburn menu-bar companion app — currently buggy upstream, run only if you want it (needs the codeburn pnpm package)" + }, { "name": "mac-mouse-fix", "tags": ["desktop-utility"], @@ -722,6 +732,22 @@ "optional": true, "description": "OCR engine — extract text from images and scanned documents" }, + { + "name": "fio", + "tags": ["benchmarking"], + "package_manager": { "macos": "brew" }, + "priority": "low", + "optional": true, + "description": "Storage I/O benchmark — sequential and random IOPS; used by macOS/benchmarks/" + }, + { + "name": "hyperfine", + "tags": ["benchmarking"], + "package_manager": { "macos": "brew" }, + "priority": "low", + "optional": true, + "description": "Command-line benchmarking tool with warmup, N runs, and JSON export" + }, { "name": "smartmontools", "tags": ["system-monitoring"], @@ -730,6 +756,58 @@ "optional": true, "description": "Monitor SSD/HDD health via S.M.A.R.T. data — use: smartctl -a /dev/sdX" }, + { + "name": "stress-ng", + "tags": ["benchmarking"], + "package_manager": { "macos": "brew" }, + "priority": "low", + "optional": true, + "description": "CPU/memory stress tester — STREAM memory bandwidth; used by macOS/benchmarks/" + }, + { + "name": "geekbench", + "tags": ["benchmarking"], + "package_manager": { "macos": "brew-cask" }, + "priority": "low", + "optional": true, + "description": "Geekbench 6 — standardized CPU + Metal GPU benchmark; used by macOS/benchmarks/standardized.sh" + }, + { + "name": "geekbench-ai", + "tags": ["benchmarking"], + "package_manager": { "macos": "brew-cask" }, + "priority": "low", + "optional": true, + "description": "Geekbench AI — standardized ML inference benchmark (Core ML / Metal / ANE); used by macOS/benchmarks/standardized.sh" + }, + { + "name": "cinebench", + "tags": ["benchmarking"], + "package_manager": { "macos": "custom" }, + "install_command": "bash \"$SETUP_ROOT/macOS/install-cinebench.sh\"", + "handled_by_setup": true, + "priority": "low", + "optional": true, + "description": "Cinebench — native Apple Silicon CPU + GPU render benchmark; used by macOS/benchmarks/standardized.sh. Installed from Maxon's .dmg (the brew cask's pinned checksum goes stale against Maxon's rolling build)" + }, + { + "name": "blender-benchmark", + "tags": ["benchmarking"], + "package_manager": { "macos": "brew-cask" }, + "priority": "low", + "optional": true, + "description": "Blender Benchmark — Metal GPU render workload (samples/min); used by macOS/benchmarks/standardized.sh" + }, + { + "name": "omlx", + "tags": ["local-llm"], + "package_manager": { "macos": "custom" }, + "install_command": "bash \"$SETUP_ROOT/macOS/install-omlx-app.sh\"", + "handled_by_setup": true, + "priority": "low", + "optional": true, + "description": "oMLX menu-bar app — native macOS front end + MLX inference server (web chat at /admin/chat). Installed from the GitHub .dmg; do not also install the omlx brew formula (port/state conflict)" + }, { "name": "nmap", "tags": ["networking"], diff --git a/platforms/macos.sh b/platforms/macos.sh index fb1cfe8..c934ce0 100755 --- a/platforms/macos.sh +++ b/platforms/macos.sh @@ -2,71 +2,87 @@ # macOS quirks: Homebrew bootstrap, brew/brew-cask/custom installs, # expat-pinned pyenv build (Tahoe fix), App Store reminders. -# Populated by brew_install_tier / brew_cask_install_tier; printed at the end. +# Populated by mac_install_list / mac_custom_install_tier; printed at the end. BREW_FAILURES=() BREW_TOTAL=0 -brew_install_tier() { - local priority="$1" names name err reason - names=$(pkg_names brew "$priority") +# mac_install_list — shared [i/N] install loop: prints +# progress, runs " " per package, collects failures into +# BREW_FAILURES instead of aborting the tier. +mac_install_list() { + local names="$1" name err reason i total label + shift + local -a cmd=("$@") list + label="${cmd[*]}" [[ -z "$names" ]] && return 0 - for name in $names; do + read -ra list <<< "$names" + total=${#list[@]} + i=0 + for name in "${list[@]}"; do + i=$((i + 1)) if [[ "$DRY_RUN" == true ]]; then - printf ' [dry-run] brew install %s\n' "$name" + printf ' [dry-run] [%d/%d] %s %s\n' "$i" "$total" "$label" "$name" continue fi + printf ' [%d/%d] %s %s...\n' "$i" "$total" "$label" "$name" BREW_TOTAL=$((BREW_TOTAL + 1)) - if ! err=$(brew install "$name" 2>&1); then + if ! err=$("${cmd[@]}" "$name" 2>&1); then reason=$(printf '%s\n' "$err" | grep -m1 'Error:' | sed 's/^Error: //') [[ -z "$reason" ]] && reason=$(printf '%s\n' "$err" | tail -1) - BREW_FAILURES+=("brew/$name: $reason") - printf ' [FAIL] brew install %s — %s\n' "$name" "$reason" + BREW_FAILURES+=("$label $name: $reason") + printf ' [FAIL] %s %s — %s\n' "$label" "$name" "$reason" fi done } -brew_cask_install_tier() { - local priority="$1" names name err reason - names=$(pkg_names brew-cask "$priority") - [[ -z "$names" ]] && return 0 - # --adopt: take ownership of apps already in /Applications (manual installs) - # instead of hard-failing the whole tier (#31) - for name in $names; do - if [[ "$DRY_RUN" == true ]]; then - printf ' [dry-run] brew install --cask --adopt %s\n' "$name" - continue - fi - BREW_TOTAL=$((BREW_TOTAL + 1)) - if ! err=$(brew install --cask --adopt "$name" 2>&1); then - reason=$(printf '%s\n' "$err" | grep -m1 'Error:' | sed 's/^Error: //') - [[ -z "$reason" ]] && reason=$(printf '%s\n' "$err" | tail -1) - BREW_FAILURES+=("cask/$name: $reason") - printf ' [FAIL] brew install --cask %s — %s\n' "$name" "$reason" - fi - done -} +brew_install_tier() { mac_install_list "$(pkg_names brew "$1")" brew install; } + +# --adopt: take ownership of apps already in /Applications (manual installs) +# instead of hard-failing the whole tier (#31) +brew_cask_install_tier() { mac_install_list "$(pkg_names brew-cask "$1")" brew install --cask --adopt; } mac_custom_install_tier() { - local priority="$1" - # shellcheck disable=SC2016 - jq -r --arg plat "$PLATFORM" --arg pr "$priority" \ - --arg w "$INCLUDE_WORK" --arg p "$INCLUDE_PERSONAL" \ - "$CORE_JQ_DEFS"'.[] | select( - .package_manager[$plat] == "custom" and prfor($plat) == $pr and - envok($plat; $w; $p) and (icfor($plat) != null) and .name != "nvm" and tagok($plat) - ) | icfor($plat)' "$PACKAGES_JSON" | + local priority="$1" cmd + # Collect failures like the brew tiers (#31) — one failed installer must not + # abort the rest of the run under set -e. Process substitution (not a pipe) + # keeps BREW_FAILURES in this shell. while read -r cmd; do - run_eval "$cmd" - done + [[ -z "$cmd" ]] && continue + [[ "$DRY_RUN" == false ]] && BREW_TOTAL=$((BREW_TOTAL + 1)) + if ! run_eval "$cmd"; then + BREW_FAILURES+=("custom: $cmd") + printf ' [FAIL] %s\n' "$cmd" + fi + done < <( + # shellcheck disable=SC2016 + jq -r --arg plat "$PLATFORM" --arg pr "$priority" \ + --arg w "$INCLUDE_WORK" --arg p "$INCLUDE_PERSONAL" \ + "$CORE_JQ_DEFS"'.[] | select( + .package_manager[$plat] == "custom" and prfor($plat) == $pr and + envok($plat; $w; $p) and (icfor($plat) != null) and (.handled_by_setup == true) + and .name != "nvm" and tagok($plat) + ) | icfor($plat)' "$PACKAGES_JSON" + ) } -mac_pipx_install_tier() { - local priority="$1" names name - names=$(pkg_names pipx "$priority") - [[ -z "$names" ]] && return 0 - for name in $names; do - run pipx install "$name" - done +mac_pipx_install_tier() { mac_install_list "$(pkg_names pipx "$1")" pipx install; } + +# Cache sudo credentials once up front. Homebrew's cask/pkg installers each shell +# out to `sudo`, so without this a fresh install prompts for the password ~6 +# times. Prime the timestamp once, then refresh it in the background until this +# script exits so every later sudo call reuses it silently. +mac_prime_sudo() { + if [[ "$DRY_RUN" == true ]]; then + printf ' [dry-run] sudo -v (cache credentials + background keepalive)\n' + return 0 + fi + printf '==> Caching credentials (you may be prompted for your password once)...\n' + sudo -v || return 0 + # || true: the subshell inherits set -e, and one failed refresh (timestamp + # revoked mid-run) must not silently kill the keepalive. stdout is redirected + # so a piped run (setup.sh | tee) sees EOF at exit instead of hanging on the + # fd this subshell holds for up to 60s. + ( while true; do sudo -n true || true; sleep 60; kill -0 "$$" 2>/dev/null || exit; done ) >/dev/null 2>&1 & } print_app_store_reminders() { @@ -84,6 +100,10 @@ print_app_store_reminders() { platform_main() { # shellcheck disable=SC2034 # consumed by deploy_zshrc in lib/core.sh CONFIG_SRC_DIR="$SETUP_ROOT/macOS" + + # ── sudo priming ────────────────────────────────────────────────────────────── + mac_prime_sudo + # ── Homebrew ──────────────────────────────────────────────────────────────── if ! command -v brew &>/dev/null; then printf '==> Installing Homebrew...\n' @@ -220,9 +240,9 @@ platform_main() { export PATH="$PNPM_HOME/bin:$PATH" if [[ "$DRY_RUN" == true ]] || command -v pnpm &>/dev/null; then pnpm_install_tier medium - - # Post-install: codeburn menubar (macOS native Swift app) - run codeburn menubar || printf 'warning: codeburn menubar failed (non-fatal)\n' >&2 + # codeburn's menu-bar app is buggy upstream, so it is NOT installed + # automatically — the codeburn-menubar packages.json entry surfaces it + # as a manual reminder instead. else printf ' pnpm not found — skipping (run corepack enable)\n' fi @@ -249,6 +269,10 @@ platform_main() { print_app_store_reminders none [[ "$INCLUDE_OPTIONAL" == true ]] && print_app_store_reminders low + # Custom entries the engine does not run (handled_by_setup != true) — + # without this they are neither installed nor surfaced (linux_main has it) + custom_reminders_section + printf '\n' printf 'Optional — run these from the repo root as needed:\n' print_related_scripts diff --git a/scripts/validate-packages.sh b/scripts/validate-packages.sh index 01d5587..6b58ddd 100755 --- a/scripts/validate-packages.sh +++ b/scripts/validate-packages.sh @@ -39,12 +39,13 @@ errors=$(jq -r ' def PLATFORMS: ["macos","ubuntu","arch","server"]; def TIERS: ["high","medium","low","none"]; def ENVS: ["work","personal"]; - def TAGS: ["ai-coding","browser","cloud-storage","communication","containers", - "database","desktop-utility","development","entertainment","local-llm", - "media","networking","photos","productivity","science","security", - "system-monitoring","terminal"]; + def TAGS: ["ai-coding","benchmarking","browser","cloud-storage","communication", + "containers","database","desktop-utility","development","entertainment", + "local-llm","media","networking","photos","productivity","science", + "security","system-monitoring","terminal"]; def prfor($p): (.priority | if type == "object" then .[$p] else . end); def optfor($p): (.optional | if type == "object" then .[$p] else . end); + def icfor($p): (.install_command | if type == "object" then .[$p] else . end); ( group_by(.name)[] | select(length > 1) | "duplicate name \"\(.[0].name // "(unnamed)")\" — \(length) entries" ), @@ -86,6 +87,17 @@ errors=$(jq -r ' then "\($nm): optional for \"\($p)\" is \($o | tojson) — must be boolean" else empty end)) else empty end), + (.handled_by_setup as $h + | if $h == null or ($h | type) == "boolean" then empty + else "\($nm): handled_by_setup is \($h | tojson) — must be boolean (the engine tests == true)" end), + + (if (.package_manager | type) == "object" + then ((.package_manager | to_entries[]) as $kv + | if $kv.value == "custom" and (icfor($kv.key) | type) != "string" + then "\($nm): custom on \"\($kv.key)\" needs a string install_command (got \(icfor($kv.key) | tojson))" + else empty end) + else empty end), + (.environment as $env | if $env == null then empty elif ($env | type) == "array"