Repository navigation
guest/process: free the process slot before the exit is visible - #127
Open
Eric Juta (ericjuta) wants to merge 1 commit into
Open
Eric Juta (ericjuta) wants to merge 1 commit into
Eric Juta (ericjuta) wants to merge 1 commit into
Conversation
reap published the exit (state, done channel, output notify) before it decremented the active count. A client that saw a process exit and started another at once could get ResourceExhausted, although a slot was about to be free. TestConcurrencyLimiter does exactly that and fails this way on a loaded machine. Decrement the active count before the exit state is published. Signed-off-by: Eric Juta <ericjohnjuta@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Tracker.reappublishes a process exit (state,donechannel, output notify) and only then callsdecrementActive(). A client that sees the exit and starts a replacement at once can get:although the slot is about to be free.
TestConcurrencyLimiterdoes exactly this (kill job 1, wait for its exit, start another), so it is flaky under load. I hit it in a Nix sandbox build on a GitHub-hosted x86_64 runner, atservice_test.go:479.Change
Call
decrementActive()before the exit state is published. No other behaviour changes: the process has already been waited and its spool files closed at that point.Evidence
time.Sleep(5 * time.Millisecond)betweenstate.output.notify()anddecrementActive()onmain(56424de):go test ./guest/process -run TestConcurrencyLimiter -count=5fails 5 of 5 with the error above.go test ./guest/...passes,go test ./guest/process -count=3passes,go vetandgofmtare clean.I did not add a test, because
TestConcurrencyLimiteralready covers the behaviour; it only lost the race rarely.