Repository navigation
fix: Fix three deadlocks with coexisting sessions - #971
Merged
rphmeier merged 3 commits intoNov 5, 2026
Merged
Conversation
A session's reverse-delta worker runs until the session is finished or dropped, but it ran on the 2-thread `rollback-worker` pool. With three coexisting sessions the third worker never started, and that session's `finish` blocked forever in `ReverseDeltaBuilder::finalize` waiting on it, although coexisting sessions are allowed. Spawn the worker on a dedicated thread instead (`spawn_thread_task`, the thread counterpart of `spawn_task`) and drop the pool. This costs one thread spawn per session when rollback is enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With `warm_up` enabled, a session's warm-up task runs until the session finishes, but it ran on the `nomt-commit` pool, which has `commit_concurrency` threads. Once that many sessions coexisted, a later session's warm-up never started, and its `finish` blocked forever joining it (with the default `commit_concurrency(1)`, two sessions suffice). Spawn the warm-up on a dedicated thread, leaving the pool to the update workers. This costs one thread spawn per session when warm-up is enabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
parking_lot's RwLock is fair: once a commit (or rollback) queues for the write side of `access_lock`, new readers wait behind it. The commit itself waits for every live session, so a thread that holds a session and opens another one (or calls `Nomt::read`) deadlocked whenever a commit was queued in between. Track live sessions per creating thread. A thread that has one takes the read lock recursively, past the queued writer; every other thread still queues behind it. Taking every session's lock recursively also fixes the deadlock, but lets overlapping sessions from other threads starve commits: with 4 threads each opening sessions of 64 reads in a loop, commits fell from ~630/s to ~0.6/s with multi-second latency. A session moved to another thread still counts for the thread that created it, so a thread holding a session it didn't create can still deadlock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
begin_sessiondocuments that multiple sessions may coexist, but three patterns of coexisting sessions deadlock. Each commit fixes one and adds a regression test that hangs on master.Found by #970
1. Rollback: a third live session can't finish
Each session's reverse-delta worker runs until the session is finished or dropped, but it ran on the 2-thread
rollback-workerpool. With three live sessions the third worker never starts, and that session'sfinishblocks forever inReverseDeltaBuilder::finalize.Fix: the worker runs on a dedicated thread (
task::spawn_thread_task, the thread counterpart ofspawn_task), and the pool is removed.2. Warm-up: a session can't finish while others are live
With
warm_upenabled, a session's warm-up task runs until the session finishes, but it ran on thenomt-commitpool (commit_concurrencythreads). Once that many sessions are live, a later session's warm-up never starts and itsfinishblocks forever joining it. With the defaultcommit_concurrency(1), two sessions are enough.Fix: the warm-up runs on a dedicated thread, leaving the pool to the update workers.
3. A thread holding a session opens another while a commit is queued
parking_lot's
RwLockis fair: once a commit or rollback queues for the write side ofaccess_lock, new readers wait behind it. The commit waits for every live session, so a thread that holds a session and opens a second one (or callsNomt::read) deadlocks.Fix: live sessions are counted per creating thread. A thread that already has one takes the read lock recursively, past the queued writer; every other thread still queues behind it. Making every session's read recursive also removes the deadlock, but it lets overlapping sessions from other threads starve commits (numbers below). Known limitation: a session moved to another thread still counts for the thread that created it, so a thread holding a session it didn't create can still deadlock this way.
Cost
begin_session → finish → drop, rollback on¹ Measured with the rejected recursive-read variant of fix 3. Benchtop opens one session at a time on one thread, so the final version takes the same fair-lock path there as master.
² Separate run, hence the different master baseline.
Testing
nomt/tests/coexisting_sessions.rs:many_coexisting_sessions_with_rollbackandmany_coexisting_sessions_with_warm_up(4 live sessions, finish the newest).nomt/src/lib.rs:session_opens_while_commit_waits_for_another_session(deadlock 3) andsession_on_another_thread_waits_for_queued_commit(commits are not starved by sessions on other threads). Both wait for the commit to queue by pollingtry_read; there are no sleeps.cargo test -p nomtpasses in release and debug, and the loom tests (--cfg loom) pass.🤖 Generated with Claude Code