Skip to content

guest/process: free the process slot before the exit is visible - #127

Open
Eric Juta (ericjuta) wants to merge 1 commit into
agent-substrate:mainfrom
ericjuta:fix/free-slot-before-exit
Open

Eric Juta (ericjuta) wants to merge 1 commit into
agent-substrate:mainfrom
ericjuta:fix/free-slot-before-exit

Conversation

@ericjuta

Copy link
Copy Markdown

Problem

Tracker.reap publishes a process exit (state, done channel, output notify) and only then calls decrementActive(). A client that sees the exit and starts a replacement at once can get:

ResourceExhausted: maximum concurrent processes limit reached (2)

although the slot is about to be free. TestConcurrencyLimiter does exactly this (kill job 1, wait for its exit, start another), so it is flaky under load. I hit it in a Nix sandbox build on a GitHub-hosted x86_64 runner, at service_test.go:479.

Change

Call decrementActive() before the exit state is published. No other behaviour changes: the process has already been waited and its spool files closed at that point.

Evidence

  • The race is rare on an idle machine. To show the window is real, I put a time.Sleep(5 * time.Millisecond) between state.output.notify() and decrementActive() on main (56424de): go test ./guest/process -run TestConcurrencyLimiter -count=5 fails 5 of 5 with the error above.
  • With this change: go test ./guest/... passes, go test ./guest/process -count=3 passes, go vet and gofmt are clean.

I did not add a test, because TestConcurrencyLimiter already covers the behaviour; it only lost the race rarely.

reap published the exit (state, done channel, output notify) before it
decremented the active count. A client that saw a process exit and
started another at once could get ResourceExhausted, although a slot was
about to be free. TestConcurrencyLimiter does exactly that and fails this
way on a loaded machine.

Decrement the active count before the exit state is published.

Signed-off-by: Eric Juta <ericjohnjuta@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant