Skip to content

perf(mpmc): back bounded queues with a ring buffer - #382

Open
ariesdevil wants to merge 1 commit into
apache:mainfrom
ariesdevil:dev_ariesdevil
Open

ariesdevil wants to merge 1 commit into
apache:mainfrom
ariesdevil:dev_ariesdevil

Conversation

@ariesdevil

@ariesdevil ariesdevil commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Partial related: #298

Summary

Back the bounded MPMC queue (added in #265) with a preallocated ring of stamped slots, adapted from crossbeam-channel's array flavor, instead of a VecDeque behind one mutex. Sends, receives, and reservations no longer take a lock unless a sender or receiver has to wait. Pending send and reserve operations still receive capacity in wait-queue order, permits still hold capacity without a slot, and cancellation and disconnection behave as before. Storage for the full capacity is now allocated when a queue is created, capacity is limited to 2,097,151, and the crate now requires targets with 64-bit atomics.

The unbounded queue keeps its mutex and drops the bounded-only code. CachePadded is copied from crossbeam-utils, and LICENSE, licenserc.toml, and .asf.yaml record the incorporated crossbeam portions.

The benchmarks gain a crossbeam-channel adapter, a spinning batch that isolates queue contention from wakeups, a capacity sweep, the 1P/8C topology, and a reservation micro-benchmark.

Design Notes

  • While no capacity is reserved and no sender is queued, the slots are the capacity, as in crossbeam. Permits and grants hold capacity without a slot. Their count, and a bit marking queued senders, sit in tail between the slot index and the lap, so every exchange on tail compares the reservation state it was computed from. The other values a sender reads, head and the slot stamps, only advance, so reading them early can only understate free capacity.
  • An exchange prepared from an earlier tail can succeed only after the lap count wraps. Positions are 64 bits, so that takes 2^54 sends at capacity 64, and limiting capacity to 2,097,151 keeps it at 2^40 sends or more. A 32-bit cursor would leave as few as two lap bits, so a stalled send could overwrite a queued value after 65,532 sends; the crate therefore supports x86, x86-64, ARMv6 or later, and AArch64, and fails to compile on targets without 64-bit atomics.
  • A permit's send claims its slot and returns its reservation in one exchange. try_reserve and dropping an unused permit take no lock unless senders are queued, and granting the last queued sender clears the queued bit in the same update. A grant removes the waiter's node and sets a flag in its pinned future, so the woken sender claims its permit without the lock.
  • While senders are queued, a send that uses a permit takes over granting the capacity that receivers free until it finishes. Without this, a single receiver that grants and wakes every queued sender itself can fall behind them, keeping them queued so that every send waits; on Linux, blocking 8P/1C then ran at half speed in some rounds.
  • Receivers leave the wait queue before taking a value, so a notification cannot be assigned to a receiver that has already taken another value.
  • Three earlier designs were measured and dropped. Taking capacity from the internal semaphore, and separate sender and receiver capacity counters, made every message read or write a counter that the other side also writes, which dominated perf profiles on x86. Keeping the count in its own word behind a version in tail was incorrect: 2^24 reservations wrap the version, after which a stalled send can claim capacity that a permit holds. A regression test covers that interleaving.

Benchmarks

Changes against ecc1780, from the medians of per-run Divan medians over rotated rounds, 10 on Linux and 6 on macOS, with rustc 1.99.0. Negative changes are faster. Each batch moves 16,384 messages, and capacity is 64 unless noted.

  • Linux: Intel Xeon 6982P-C, 16 cores, KVM guest, Ubuntu 26.04, kernel 7.0.
  • macOS: Apple M5 Pro. Three asyncband cells had an interquartile range of 19% to 29%; the rest stayed within 10%.
ecosystem mpmc::bounded Linux macOS
thread_batch −49% to −94%; 1P/8C +15% −40% to −75%; 1P/1C +3%
thread_batch_capacity 1024 −37% to −93%; 1P/8C +16% −2% to −82%; 8P/8C +7%
thread_batch_capacity 1 +2% to +4% +3% to +19%
spin_batch −64% to −95%; 8P/8C +8% −31% to −79%
task_batch, current-thread −19% to −32% −11% to −17%
task_batch, 4 workers −24% to −73% −35% to −8%, except 1P/1C +8%, 1P/4C +22%, 1P/8C +10%, 4P/1C +4%
primitives mpmc::bounded Linux macOS
try_send_then_try_recv 27.1 → 13.7 ns (−49%) 9.3 → 4.9 ns (−47%)
send_then_recv 31.8 → 18.4 ns (−42%) 12.7 → 8.2 ns (−36%)
try_reserve_then_send 39.0 → 31.1 ns (−20%) 13.3 → 6.5 ns (−51%)
wake_blocked_sender 85.2 → 94.5 ns (+11%) 28.1 → 30.9 ns (+10%)

On Linux, spin_batch takes 0.49x to 0.97x of crossbeam-channel's time, and blocking 2P/2C and 8P/8C take 0.68x and 0.47x. Blocking 1P/8C takes 3.2x and capacity 1 takes 3.3x to 11x, because FutureExt::block_on parks without spinning first. The remaining regressions are on paths where every operation waits: capacity 1, a blocked sender's handoff, and 1P/8C, where the producer wakes a parked consumer for most messages. On macOS, a multi-thread Tokio runtime with one producer is also slower.

The unbounded queue changed by −6% to +2%. Competitor controls in the same binaries moved by a median of −0.1% on Linux and −0.2% on macOS, and 102 of 112 Linux and 95 of 112 macOS control cells stayed within ±5%.

@ariesdevil ariesdevil changed the title perf(mpmc): make bounded queues lock-free perf(mpmc): back bounded queues with a ring buffer Oct 20, 2026
@ariesdevil
ariesdevil force-pushed the dev_ariesdevil branch 3 times, most recently from 2e2ded8 to 94d174b Compare October 21, 2026 07:31
Replace the bounded queue's VecDeque behind one mutex with a preallocated ring of stamped slots adapted from crossbeam-channel's array flavor. Sends, receives, and reservations take no lock unless a sender or receiver has to wait, while wait-queue order, exact capacity, permits, cancellation, and disconnection behave as before. Storage for the full capacity is allocated when the queue is created.

The capacity that permits and grants hold, and a bit marking queued senders, live in the tail cursor beside the position, so every exchange on it compares the reservation state it was computed from. While senders are queued, a send with a permit grants the capacity that receivers free until it finishes, so a single receiver does not have to wake every queued sender itself.

Positions are 64 bits, and capacity is limited to 2,097,151, so a stalled exchange can be fooled only after at least 2^40 sends. A 32-bit cursor would leave as few as two lap bits, so the crate now requires 64-bit atomics and supports x86, x86-64, ARMv6 or later, and AArch64.

Copy CachePadded from crossbeam-utils and record the incorporated crossbeam portions in LICENSE, licenserc.toml, and .asf.yaml.

Add benchmarks against crossbeam-channel: a spinning batch that isolates queue contention from wakeups, a capacity sweep, the 1P/8C topology, and a reservation micro-benchmark. Split the bounded adapter trait so channels without async operations can join the thread workloads.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant