collectivex

CollectiveX - Experimental (Beta)

CollectiveX is an experimental MoE expert-parallel communication benchmark. It measures dispatch, combine and paired roundtrip latency across EP libraries and accelerator systems, and writes one result file per case into results/ on the cluster that ran it. A second suite, the vLLM swap_blocks benchmark, measures pinned CPU↔GPU and same-GPU block copies through the same matrix and pool launchers.

CollectiveX runs through the CollectiveX Sweep workflow or by hand on each pool (see Running By Hand). Uploading the results to the CollectiveX database is the publish step: the dashboard shows the latest stored run. Results are not committed to this repository. CollectiveX does not rank, recommend, or select; any downstream display or comparison is the consumer's responsibility. docs/methodology.md states the measurement rules in a summary, including each backend's engine contract, and then defines each published field.

Coverage

Systems	EP8	EP16
H100/H200/B200/B300	1x8 NVLink, scale-up	2x8 NVLink + RDMA, scale-out
MI300X/MI325X/MI355X	1x8 XGMI, scale-up	2x8 XGMI + RDMA, scale-out
GB200/GB300	2x4 MNNVL, scale-up	4x4 MNNVL, scale-up

Physical host count does not determine scope: both GB topologies stay inside one 72-GPU MNNVL scale-up domain. configs/platform_config.json registers which backend runs where. normal mode runs the decode and prefill ladders; low-latency runs decode, plus prefill for NCCL EP.

Backend	Engine selector (maturity)	normal	low-latency
deepep-v2	vLLM --all2all-backend deepep_v2, SGLang --moe-a2a-backend deepep (production)	EP8, EP16: H100, H200, B200, B300, GB200, GB300	EP8: H100. EP8, EP16: H200, B200, GB200, GB300
mori	vLLM --all2all-backend mori_*, SGLang --moe-a2a-backend mori (production)	EP8, EP16: MI300X, MI325X, MI355X	EP16: MI300X, MI325X, MI355X
uccl-ep	none; engines load it as a drop-in deep_ep under vLLM's DeepEP backends (candidate)	EP8: H100, H200, B200, MI300X, MI325X, MI355X. EP16: H100, H200, B200, MI300X, MI325X	EP8: B200. EP8, EP16: H100, H200. EP16: MI300X, MI325X
nccl-ep	TensorRT-LLM NcclEP (production for low-latency rank-major BF16 where it matches, candidate otherwise)	EP8, EP16: H100, H200, B200, B300, GB200, GB300	EP8: B300. EP8, EP16: H100, H200, B200, GB200, GB300
flashinfer-ep	vLLM --all2all-backend flashinfer_nvlink_one_sided (production)	EP8: H100, H200, B200, B300. EP8, EP16: GB200, GB300	none

Every backend runs a BF16 dispatch control and an FP8 dispatch; nccl-ep's FP8 is its internal DS_FP8E3M4 recipe, low-latency prefill and decode. Combine is always BF16. Coverage is uniform routing only.

Eleven of the registered EP cases are unmeasured in the published run: MI355X MoRI EP16, H100 EP16 prefill for DeepEP V2 and NCCL EP low-latency, and MI325X UCCL-EP low-latency EP16 FP8. docs/methodology.md gives each reason.

results/cases.json is the committed copy of the full matrix: 282 requested cases, of which 254 are runnable EP cases, 18 are swap-blocks cases (two per pool) and 10 are unsupported EP cells (UCCL-EP EP16 on MI355X and FlashInfer EP16 on H100, H200, B200 and B300, once per phase; FlashInfer's MNNVL workspace needs every rank inside one NVLink domain). A cell is listed as unsupported only when its backend runs on that SKU at another EP size, and low-latency adds only runnable cells, so a missing low-latency cell (DeepEP V2 on B300, for example) appears nowhere; docs/methodology.md gives each reason. The NCCL EP contract, row identity and wheel pin are in docs/nccl-ep.md.

Running In CI

Dispatch the CollectiveX Sweep workflow, choosing a pool (or all) and the same filters run.sh takes. Each shard becomes one job in the InferenceX CI priority queue, scored through inferencex-e2e/configs/ci-priority.yaml, where collectivex ranks above every engine benchmark except a model that file boosts further (currently kimik3.1). The job lands on the pool's runner (runner_label and runner_partition in configs/platform_config.json), runs run.sh --shard and uploads results/<sku>/ as an artifact to publish as below. Hand runs bypass the queue and wait for whatever CI holds the nodes.

Running By Hand

Run from the pool's Slurm login node, in a checkout of the commit you want to measure:

collectivex/run.sh h200-dgxc --list-shards          # what will run, one allocation per shard
setsid nohup collectivex/run.sh h200-dgxc > run-h200.log 2>&1 < /dev/null &
collectivex/run.sh h200-dgxc --shard h200-dgxc-nccl-ep-bf16-n2   # rerun one failed shard

run.sh resolves the pool's cases with sweep_matrix.py (--suite, --backend, --ep, --modes and --precisions narrow it, and --swap-profile picks the swap-blocks grid), gives each shard (one backend, mode, precision and node count) its own allocation through the pool's launcher, and copies every result to results/<sku>/<case_id>.json, replacing the previous result for that case. A failed shard does not stop the others; the closing summary lists each shard's exit code and runs summarize.py over the pool's results. Interrupting run.sh releases its allocations. Runtime logs stay under /tmp/inferencex-collectivex-<uid>/logs/<execution>/ on the login node.

h200-dgxc, b200-nscale and gb200 run from the registry alone. h100-dgxc, b300, gb300 and the AMD pools also need an operator document that names a stage directory (see Runner Configuration).

These variables change what a row measures. The result records each one (in kernel_generation, implementation.fp8_consume, implementation.rank_alignment or implementation.library_config), but the case_id does not, so a run with one replaces the case's stored default-configuration result:

Variable	Default	Effect
COLLX_CUDA_GRAPH=0	1	times graph-capable decode cells eager (-eager-spin-ar)
CX_FP8_CONSUME=dequant	native	stages every FP8 pair inside the chain, a verification hatch
COLLX_NCCL_LL_LAYOUT=expert-major	rank-major	runs nccl-ep low-latency on its expert-major layout
COLLX_NCCL_LL_ZERO_COPY=0	1	stages nccl-ep low-latency dispatch, TensorRT-LLM's graphed path before #20631 (drops -zc; a library row)
COLLX_ALIGN_SPIN_US	2000	the per-sample rank-alignment spin

COLLX_TIME (allocation minutes: 300 per EP shard and 45 per swap-blocks shard unless set), COLLX_RUN_TIMEOUT (the per-case hang guard, 5400 s), COLLX_IMAGE_REFRESH=1 (re-import the image) and COLLX_ALLOCATION=sbatch change only how a shard runs. COLLX_ALLOCATION=sbatch holds the nodes with a batch job that only sleeps instead of salloc. It is for a pool whose controller kills interactive allocations about ten seconds after granting them while batch jobs run; every later step already goes through srun --jobid.

Results record the commit they were measured at, so run.sh refuses to start while tracked files under collectivex/ (outside results/) have uncommitted edits. --allow-dirty runs anyway and marks the results source_dirty, so they cannot be published. results/<sku>/ is gitignored. To publish, copy the pool's results/<sku>/ and results/cases.json off the cluster and, from an InferenceX-app checkout with DATABASE_COLLECTIVEX_WRITE_URL set, run bun run admin:db:ingest:collectivex --dir <copy>/results. It lays the uploaded files over the latest stored run, replacing those cases, and stores the result as a new run. It rejects a file that is not a runnable case in cases.json, was measured from uncommitted code, or contains an IP address or host-identifying field, and every uploaded file must share one source SHA.

Changelog

Every PR that changes what CollectiveX measures appends one entry (or one per scope) to the end of perf-changelog.yaml: the date, the PR link, the pools it affects and, optionally, the suites, backends, modes, phases, EP sizes and precisions that narrow it, with a description. python3 collectivex/changelog.py validate checks the file, and CI also checks that existing entries are unchanged and that new entries link the PR adding them, name registry pools and select runnable cases with every selector value.

An upload names the merged PRs it publishes: admin:db:ingest:collectivex --dir <copy>/results --pr <n>. The database stores those entries against the new run, so each run records the changes it measures and each change the runs that measured it, with the before and after results of every case it selects.

results/cases.json stays committed; a unit test fails when it no longer matches the configs, so regenerate it whenever they change:

python3 collectivex/sweep_matrix.py --suites ep,swap-blocks --cases-json collectivex/results/cases.json

Runner Configuration

Each SKU's Slurm, storage, image and network values come from its entry in configs/platform_config.json, which also records architecture and product, fixed placement, launcher, runnable backend/EP pairs and the scale-out fabric identity (NIC and switch, so same-GPU clusters on different fabrics are distinct entries). An optional operator document at COLLECTIVEX_OPERATOR_CONFIG, or else ${XDG_CONFIG_HOME:-$HOME/.config}/inferencex/collectivex.json, overrides the entry's operator block field by field:

{"runners": {"b300": {"storage_roots": ["/data/home/<you>"]}}}

It accepts partition, account, qos, squash_dir, stage_dir, storage_roots, enroot_cache_path, exclude_nodes, nodelist, lock_dir and the network fields below. An unknown field, a SKU the registry lacks and non-JSON input fail closed, and no value is ever evaluated as shell. Duplicate JSON keys are not rejected (json.load keeps the last), and entries for other SKUs are not validated, so a misspelled SKU is silently ignored. The registry's network block is applied after the document and wins for every key it sets.

stage_dir must be visible at the same path on the login node and every allocated node. Each shard copies collectivex/ into a mode-0700 job_<execution> child there and removes it once its allocation is confirmed released. Without one, the stage goes beside the image under squash_dir, which h200-dgxc and b200-nscale do; h100-dgxc, b300, both GB pools and the AMD pools require one, and gb200's registry storage_roots supplies it. storage_roots derives both stage_dir and squash_dir (<root>/collectivex/stage and <root>/collectivex/containers, from the first root where both can be created) after the document is merged in, so on gb200, whose registry entry carries storage_roots, a lone stage_dir in the document is overridden: give the document its own storage_roots there. A run under an account other than the one that owns the registry's squash_dir (a CI account's home on h100-dgxc and b300) needs its own squash_dir or storage_roots as well: each shard creates its backend cache under squash_dir, and an image squash another account owns is reused only while it is current.

Every non-MNNVL EP16 placement requires rdma_devices, the operator-approved scale-out devices. The other network fields are socket_ifname, ib_gid_index, rdma_service_level, rdma_traffic_class, rail_isolated, rdma_fabric, rdma_relaxed_ordering, rdma_gbs, uccl_rdma_recv_tokens, host_verbs_provider, deepep_efa_fork and single_node_rdma_devices. Without socket_ifname each node's default-route interface carries the cross-node sockets (h100-dgxc, h200-dgxc, b300). HCAs are never guessed, and every scale-out node must show its socket interface up and an active port on each listed device before backend setup; the methodology's Execution Isolation says what each field drives.

Enroot imports the image once per pool (once per node where the squash dir is node-local, as on mi355x) into a squash keyed by image tag and platform; the tag's manifest digest in its container registry, resolved at launch, decides when it is stale, and COLLX_IMAGE_REFRESH=1 forces a re-import. An image owned by another account is reused when readable and current, but never replaced or deleted; the launcher stops with an error instead. Backends the image does not ship (DeepEP V2 and UCCL-EP, built at their source pins in runtime/common.sh, the nccl-extensions wheels, FlashInfer at vLLM's pin, and aws-ofi-nccl where an EFA host needs it) are cached in a cluster-local backend cache that reaches the container only as /cx-cache.

Local Checks

python3 -m unittest discover collectivex/tests -p 'test_*.py'
python3 collectivex/sweep_matrix.py --backend all --out /tmp/cx-matrix.json >/dev/null
for f in collectivex/run.sh collectivex/runtime/*.sh collectivex/launchers/*.sh; do bash -n "$f"; done
collectivex/run.sh h200-dgxc --dry-run   # each shard's environment, without allocating

Core paths are run.sh, configs/, sweep_matrix.py, bench/, runtime/, launchers/, results/cases.json and tests/. summarize.py and bandwidth.py read results and gate nothing.