Skip to content

perf(channels): optimize bounded MPMC and SPMC queues - #382

Open
ariesdevil wants to merge 1 commit into
apache:mainfrom
ariesdevil:dev_ariesdevil
Open

ariesdevil wants to merge 1 commit into
apache:mainfrom
ariesdevil:dev_ariesdevil

Conversation

@ariesdevil

@ariesdevil ariesdevil commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Related: #298

Summary

Optimize bounded MPMC and SPMC queues with separate producer/consumer progress and cache-line isolation, while preserving exact logical capacity, reservation ordering, cancellation, disconnection, and endpoint traits. Unbounded queues retain their mutex-based backend.

Capacity-one queues and configurations beyond the ring allocation budget retain lazy storage. Bounded MPMC also uses that backend on targets without native 64-bit atomics. There is no new public capacity limit. These implementations are not lock-free.

Known limitation: the immediately-ready capacity-one async round trip regresses on Linux. A fixed-single-CPU follow-up measured a 44.9% increase in median elapsed time; details are below.

Design Notes

  • Cursors publish completed payload transfers, not outstanding claims. Each side protects its own cursor and payload move and caches peer progress. Cursors wrap at twice the capacity; correctness does not depend on a generation counter taking sufficiently long to wrap. There are no per-slot stamps or packed capacity bits.
  • MPMC reservations and detached grants belong to producer state, protected by the producer lock. Waiters use stable IDs instead of pointers into pending futures. A waiting send consumes its grant and publishes within one critical section, without constructing a permit and reacquiring the lock.
  • Receiver registration is separate from the consumer cursor. A receiver unregisters before attempting consumption, so notification bookkeeping cannot remain attached after it takes a value. The hot path does not nest these locks; wake callbacks and replaced/cancelled waker destruction happen outside locks.
  • SPMC keeps a specialized single-producer implementation rather than inheriting MPMC reservation machinery. Both use the platform-aware CachePadded helper. Crossbeam provenance is retained in source and LICENSE.

Benchmarks

Reference library: ae99c15 (main). Candidate: a1284ba. Both use the same expanded benchmark harness and rustc 1.99.0. Values below are median fixed-work elapsed times; negative percentages are faster, not tail-latency claims.

  • macOS: Apple M5 Pro, eight rotating runs per implementation.
  • Linux: Intel Xeon 6982P-C, KVM, kernel 7.0; six rotating runs per implementation, restricted to guest CPUs 0,2,...,30 (one thread per reported core). Revisions used independent build directories and immutable benchmark executables with verified hashes. The Linux matrix collected 414 implementation measurements.
Scenario Rounds macOS main → PR (ms) Linux main → PR (ms)
mpmc/bounded/tasks-2p2c-cap64-4w 270 299.5 → 242 (-19.2%) 1826 → 1003.5 (-45.0%)
↳ AsyncbandMpmcReserve 270 343 → 298 (-13.1%) 1935.5 → 1134.5 (-41.4%)
mpmc/bounded/threads-2p2c-cap64 120 275.5 → 182 (-33.9%) 1075.5 → 686.5 (-36.2%)
↳ AsyncbandMpmcReserve 120 318 → 216 (-32.1%) 1256 → 962.5 (-23.4%)
mpmc/bounded/threads-1p8c-cap64 27 368 → 269.5 (-26.8%) 460 → 176 (-61.7%)
mpmc/bounded/threads-2p2c-cap1 4 338.5 → 338.5 (+0.0%) 318 → 316 (-0.6%)
spmc/bounded/tasks-1p4c-cap64-4w 320 278 → 232 (-16.5%) 1072 → 687.5 (-35.9%)
spmc/bounded/tasks-1p4c-cap1024-4w 320 265.5 → 221 (-16.8%) 1538.5 → 801.5 (-47.9%)

The earlier stamped-slot revision (c766001) remains faster in several MPMC workloads. On Linux, this candidate takes 30.4% longer for the 2P2C async case and 155.2% longer for the 2P2C threaded case than that revision, while the 1P8C threaded case takes 60.7% less time. This PR does not claim to retain all of that revision's peak throughput.

Regressions and limits

  • The capacity-one primitives::mpmc::bounded::send_then_recv benchmark uses the lazy-storage fallback, not the ring. In the six-run single-CPU follow-up, main's median was 755.5 ms and the candidate's was 1095 ms for 24 million round trips: +44.9%. Main also exhibits a slower mode near 924 ms; the candidate remains slower than that mode. The regression is unresolved; async backend dispatch/future construction is a follow-up investigation, not an established cause.
  • Other Linux capacity-one MPMC microbenchmarks regress by approximately 2–4%; the SPMC capacity-one task workload regresses by 3.2%. Unbounded results are broadly unchanged and are not claimed as improvements.
  • P99/P999 latency and oversubscribed scheduling behavior have not been measured. Linux results come from one VM; these are workload-specific results, not a library-wide speedup claim.

Validation

  • cargo x test, cargo x lint, and cargo x check passed locally, along with MPMC/SPMC ecosystem smoke checks.
  • Targeted Miri: 13 ring tests passed; MPMC integration 33 passed/2 ignored; SPMC integration 30 passed/2 ignored. The powerpc-unknown-linux-gnu fallback run passed 33 tests with 2 ignored.
  • Linux cargo x test: 735 passed. Ecosystem and primitives builds and smoke checks passed for all three compared revisions. Nightly lint, the nightly feature matrix, and Miri were not run on that host because the required toolchain/tools were unavailable.
  • Coverage includes complete cursor wraps, held reservations across wraps, exactly-once delivery under contention, cancellation before payload destruction, close/wakeup behavior, reentrant waker destruction on both bounded backends, large logical capacities, and legacy endpoint traits.

@ariesdevil ariesdevil changed the title perf(mpmc): make bounded queues lock-free perf(mpmc): back bounded queues with a ring buffer Oct 20, 2026
@ariesdevil
ariesdevil force-pushed the dev_ariesdevil branch 4 times, most recently from 94d174b to 4ea81c4 Compare October 21, 2026 08:39
Comment thread licenserc.toml Resolved
Comment thread .asf.yaml Resolved
Comment thread asyncband/src/lib.rs Resolved
Comment thread asyncband/src/lib.rs Resolved
@tisonkun

tisonkun commented Nov 3, 2026

Copy link
Copy Markdown
Member

@ariesdevil You may update the branch to resolve the conflict with the main branch, and I will also adopt the new benchmark best practice that you can leverage. Once this pull request is clean, I will do another review.

Comment thread CHANGELOG.md Outdated

### Notable changes

* Bounded MPMC queues now panic when created with a capacity greater than 2,097,151.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we now have this limitation?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ring's tail packs the slot index, the disconnect mark, the capacity that permits and grants hold, the queued-senders bit, and a lap count into one 64-bit word, so every exchange on it checks all of that state at once. That is what makes a send that stalled before its exchange fail if reservations changed meanwhile; the versioned counter this replaced could be fooled after 2^24 reservations. The index and the reserved count each need about log2(capacity) bits, leaving the rest for the lap count, and a stale exchange can only succeed after the lap count wraps. Capping capacity at 2^21 − 1 keeps at least 20 lap bits, so a wrap takes at least 2^40 sends. At 2^28 it would take about 2^34 sends, which a fast producer reaches within minutes while another thread is descheduled. Larger buffers can use mpmc::unbounded.

@ariesdevil ariesdevil changed the title perf(mpmc): back bounded queues with a ring buffer perf(channels): optimize bounded MPMC and SPMC queues Nov 18, 2026
Back bounded MPMC and SPMC queues with preallocated rings that track producer and consumer progress
separately and keep each side's hot state on its own cache line. Cursors publish completed payload
transfers and cache peer progress, with no per-slot stamps or packed capacity bits. MPMC
reservations and detached grants belong to producer state under the producer lock, and waiters use
stable IDs instead of pointers into pending futures. Receiver registration is separate from the
consumer cursor, and wake callbacks and waker destruction run outside locks. SPMC keeps a
specialized single-producer ring.

Exact logical capacity, reservation ordering, cancellation, disconnection, and endpoint traits are
unchanged. Unbounded queues keep the mutex backend, which bounded queues also use at capacity one
and beyond the ring allocation budget, and bounded MPMC uses on targets without native 64-bit
atomics. Copy CachePadded from crossbeam-utils and record the incorporated crossbeam portions in
LICENSE.

The benchmarks gain reserved-send and crossbeam-channel MPMC adapters, fixed-work 1P/8C,
capacity-one, spinning, and SPMC scenarios, and reservation micro-benchmarks.

Co-authored-by: tison <wander4096@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants