Skip to content

feat(run-task): send SIGTERM to the task's processes shortly before maxRunTime - #1084

Open
benoitgoupilleau wants to merge 1 commit into
taskcluster:mainfrom
benoitgoupilleau:run-task-max-run-time-abort
Open

benoitgoupilleau wants to merge 1 commit into
taskcluster:mainfrom
benoitgoupilleau:run-task-max-run-time-abort

Conversation

@benoitgoupilleau

@benoitgoupilleau benoitgoupilleau commented Oct 30, 2026 •

Copy link
Copy Markdown

When a task exceeds its maxRunTime, the worker kills its processes straight away, so test harnesses can't log that they were aborted and their structured logs just stop.

This makes run-task abort the task itself shortly before the deadline:

  • The task transform passes maxRunTime to the task in the TASKCLUSTER_MAX_RUN_TIME environment variable, for both docker-worker and generic-worker payloads.
  • On POSIX, run-task schedules an abort at maxRunTime minus 5%, capped at 60 seconds early. It sends SIGTERM to its own process group: run-task is PID 1 under docker-worker/d2g and the group leader under generic-worker, which starts it with Setpgid.
  • run-task ignores that SIGTERM itself, through a no-op handler rather than SIG_IGN so that its children still get the signal, and keeps forwarding the task's output while the harness shuts down.
  • After an abort, run-task exits 1 whatever the command returned, whether the abort hit the checkout, the fetches or the task's command, and prints a line saying the task was aborted for exceeding its maxRunTime. A command that hasn't started by then isn't started, and a required command isn't retried.

The worker's own kill at maxRunTime is unchanged and remains the backstop if the task's processes ignore SIGTERM. Nothing changes on Windows, and tasks without TASKCLUSTER_MAX_RUN_TIME in their environment behave as before.

This is the upstream counterpart of the gecko change in https://bugzilla.mozilla.org/show_bug.cgi?id=2082559 (D338003), needed by the gecko tasks that run the upstream run-task as run-task-git.

Tests cover the margin calculation, invalid values, the SIGTERM handler, the process-group signal, a real child that receives SIGTERM while its output keeps flowing, the exit code after an abort, commands skipped after an abort, and the transform for both worker implementations.

@benoitgoupilleau
benoitgoupilleau requested a review from a team as a code owner October 30, 2026 11:26
@benoitgoupilleau
benoitgoupilleau force-pushed the run-task-max-run-time-abort branch from 544a30a to 77add46 Compare October 30, 2026 14:35

@jcristau jcristau left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes to take this out of the review queue, it needs an update to match the latest changes to the run-task-hg patch.

…axRunTime

When a task exceeds its maxRunTime, the worker kills its processes straight
away, so test harnesses can't log that they were aborted, and their
structured logs just stop.

run-task now aborts the task itself, 5% of maxRunTime early and at most 60
seconds early: it sends SIGTERM to its own process group, ignores that
signal itself, keeps forwarding the output of the running command until it
exits, and then exits 1 whatever that command returned, whether the abort
hit the checkout, the fetches or the task's command. The task transform
passes maxRunTime to run-task in the TASKCLUSTER_MAX_RUN_TIME environment
variable, for both docker-worker and generic-worker.

The command stays in run-task's process group, since that is what the
worker kills on maxRunTime, cancellation and shutdown. The worker's own
kill is unchanged, and still happens if the task's processes ignore
SIGTERM. Nothing changes on Windows.

This is the upstream counterpart of the gecko change in
https://bugzilla.mozilla.org/show_bug.cgi?id=2082559, needed by the gecko
tasks that run the upstream run-task as run-task-git.
@benoitgoupilleau
benoitgoupilleau force-pushed the run-task-max-run-time-abort branch from 77add46 to 823ee1d Compare November 5, 2026 17:34
@benoitgoupilleau

Copy link
Copy Markdown
Author

Updated to match the current D338003: run-task installs a no-op SIGTERM handler and always signals its own process group (no current_child fallback), run_command() returns 1 after an abort and doesn't start a command once the task is aborted, and retry_required_command() doesn't retry after one. Tests updated accordingly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants