Skip to content

feat: release resumable jobs after 45s and fail as AWOL only after awol_timeout (AWOL plan, phase 4) - #164

Open
prasadlohakpure wants to merge 3 commits into
mainfrom
awol/phase-4-awol-timeout
Open

prasadlohakpure wants to merge 3 commits into
mainfrom
awol/phase-4-awol-timeout

Conversation

@prasadlohakpure

@prasadlohakpure prasadlohakpure commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

AWOL-on-rollout plan

This PR is Phase 4. Phases 1 and 2 merged in #161, Phase 3 in #162.

Problem

#162 raised stale_job to 1 hour so a short database or network hiccup wouldn't fail every non-resumable job. That one setting also decides when a resumable job is released, and an hour is too late for both resumable plugins:

  • sparkeks: the SparkApplication template sets timeToLiveSeconds: 300. If the application finishes while no agent is watching, the operator deletes it after 5 minutes, and the resume an hour later submits it again.
  • ecs: ECS keeps stopped tasks for about an hour. A task that stopped early in that hour may be gone by the time the job is released, and the resume fails.

Job states and the active_jobs row

Statuses (job_statuses): new 1, accepted 2, running 3, failed 4, killed 5, succeeded 6, canceling 7, canceled 8. Nothing sets killed today.

An active_jobs row is the lease for an async job: agent_name says which agent holds it, last_heartbeat when that agent last reported in. Sync jobs never get a row; they run inside the request and are recorded in jobs afterwards, so the janitor never sees them.

Step jobs.job_status_id active_jobs row Who
Async job submitted new inserted, agent_name and last_heartbeat null API (insertJob)
Claimed by an agent new to accepted agent_name set, last_heartbeat now agent (active_select.sql, getAsyncJobs)
Plugin starts accepted to running last_heartbeat refreshed every keepalive (5 s) while the job runs agent (runAsyncJob, jobKeepalive)
Plugin finishes succeeded or failed deleted agent (updateAsyncJobStatus)
Cancel requested (any status except failed, killed, succeeded, canceled) to canceling unchanged API (status_cancel_update.sql)
Agent running the job sees canceling stays canceling deleted agent (runJob cancel monitor, then updateAsyncJobStatus)
Janitor finishes a cancel canceling to canceled deleted (if still there) janitor, after Cleanup
Janitor fails a job as AWOL accepted or running to failed deleted janitor, after Cleanup
Janitor releases a job (after stale_job, changed here from 1 h to 45 s; accepted jobs now too) unchanged (accepted or running) agent_name set to null; last_heartbeat kept janitor (active_jobs_release.sql, only if the heartbeat is still older than stale_job)
Released job claimed by another agent running stays running; accepted stays accepted agent_name set, last_heartbeat now agent; runJob calls Resume for a running job, Execute for an accepted one

A canceling job is never claimed (active_select.sql skips status 7), so a job canceled before any agent picked it up is canceled by the janitor.

Behaviour per scenario

Times are without a heartbeat. "Resumable" means the job's command plugin implements plugin.Resumer (sparkeks, ecs).

# Job Janitor action
1 running, resumable, stale for over stale_job (45 s) Released; another agent claims it and calls Resume
2 accepted, resumable, stale for over stale_job Released; another agent claims it and runs it from the start with Execute
3 accepted or running, resumable, heartbeat fresh Left alone
4 released and not yet claimed again, last heartbeat under awol_timeout ago Left alone; any live agent claims it within a second, and the janitor only runs inside a live agent
5 accepted or running, resumable, claimed or released, stale for over awol_timeout (1 h) Failed as AWOL, Cleanup called. The janitor locks the active_jobs rows, so a job an agent claims at that moment is left out
6 accepted or running, not resumable, stale for over stale_job Failed as AWOL, Cleanup called (as before #162)
7 accepted or running, not resumable, heartbeat fresh Left alone
8 canceling, no active_jobs row, or released Canceled, Cleanup called (as before)
9 canceling, claimed, stale for over stale_job Canceled, Cleanup called (as before #162)
10 canceling, claimed, heartbeat fresh Left alone; the agent running it cancels it (as before)
11 new, never claimed Left alone (as before)

A job that finishes, fails or is canceled through its agent is never selected, since the agent removes its active_jobs row.

Timing

From an agent dying, a resumable job is released within stale_job plus up to clean_interval (45 s + 60 s = under 2 minutes), and claimed by another agent within a second after that. That is inside Spark's 5-minute TTL and well inside ECS's stopped-task retention, so the resume finds the remote work.

Testing

  1. Picking up resumable (ecs/spark-eks) jobs after it goes stale (45s since last heartbeat) and before awol timeout (1 hr since last heartbeat):
    A. ECS Job started:
Screenshot 2026-11-13 at 9 32 03 AM

B. Heimdall rolls out

C. Job being stale for over 45 seconds
Screenshot 2026-11-13 at 9 38 11 AM
D. Job picked up

Screenshot 2026-11-13 at 9 38 42 AM
  1. AWOLing resumable (ecs/spark-eks) jobs after awol timeout:
    A. ECS Job started:
    B. Heimdall rolls out
    C. Job being stale for over 1 hour
    D. Job AWOLed up
Screenshot 2026-11-13 at 11 59 46 AM
  1. AWOLing non resumable jobs after it goes stale (45s since last heartbeat)
    A. Shell plugin Job started:
Screenshot 2026-11-13 at 12 07 01 PM

B. Heimdall rolls out

C. Job being stale for over 45 seconds

Screenshot 2026-11-13 at 12 07 30 PM

D. Job AWOLed up

Screenshot 2026-11-13 at 12 08 17 PM

Known gaps

Agents are only replaced during an ECS rollout. A draining agent keeps heartbeating the jobs it holds, and async jobs run on context.Background(), so at the drain deadline the process exits without writing a status. The job is then released 45 s later and resumed by a new agent. That rules out two agents watching one job, unless a database blip over 45 s cuts an agent's heartbeat while it is still running. What is still not covered:

  • A database or network blip over 45 s fails jobs of commands that can't resume, as before feat: Resume sparkeks and ecs jobs on another agent instead of failing them as AWOL (AWOL plan, phase 3) #162, and releases resumable ones while their agent may still be watching.
  • A job that crashes every agent that claims it is never failed as AWOL: each claim sets last_heartbeat to now, so awol_timeout only bounds a released job no agent picks up. Bounding crash loops needs a timestamp a claim doesn't reset, or a release counter; follow-up.
  • ECS timeout restarts on resume: polling measures timeout from the resume, so each rollout gives a running ECS job a fresh timeout.
  • ECS tasks stopped over about an hour before a resume are no longer returned by ECS, so earlier failures are undercounted and a restart can reuse a client token. Only a job running over an hour with an early task failure can hit this; token-based resume, a follow-up PR, closes it.
  • A Spark application that finished while no agent was watching is deleted 5 minutes after it ends (timeToLiveSeconds: 300 in heimdall-config), and a later resume reruns the query. A release takes under 2 minutes, so this needs no agent up for over 5 minutes.

Copilot AI balanced review requested due to automatic review settings November 12, 2026 15:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A claim/janitor race can incorrectly fail and clean up a newly reclaimed job, while cleanup calls without deadlines can delay release commits long enough to trigger duplicate submissions.

2 open issues
What changed in this PR

This implements phase 4 of the rollout recovery plan, separating prompt release of resumable jobs from the longer timeout for failing jobs as AWOL.

Changes:

  • Sets the default resumable-job release threshold to 120 seconds and adds a separate 3600-second awol_timeout.
  • Moves release-versus-fail decisions into SQL and preserves released jobs’ heartbeats so unclaimed work can eventually time out.
  • Replaces job-splitting tests with coverage for identifying resumable commands.
File Description
internal/​pkg/​janitor/​queries/​stale_and_canceling_jobs_select.sql Selects actionable jobs and determines whether to release or finalize them.
internal/​pkg/​janitor/​queries/​active_jobs_release.sql Clears ownership while retaining the heartbeat for AWOL detection.
internal/​pkg/​janitor/​job.go Uses query-provided release decisions instead of splitting jobs in Go.
internal/​pkg/​janitor/​job_test.go Removes tests for the deleted splitting helper.
internal/​pkg/​janitor/​janitor.go Adds AWOL configuration and collects resumable command IDs.
internal/​pkg/​janitor/​janitor_test.go Tests resumable-command filtering and ordering.
cmd/​heimdall/​heimdall.go Configures separate release and AWOL timeout defaults.

Review effort: Balanced

Comment thread internal/pkg/janitor/job.go Resolved
@prasadlohakpure
prasadlohakpure force-pushed the awol/phase-4-awol-timeout branch 2 times, most recently from d968a28 to 1662101 Compare November 12, 2026 17:07
@prasadlohakpure prasadlohakpure changed the title feat: release resumable jobs after 120s and fail as AWOL only after awol_timeout (AWOL plan, phase 4) feat: release resumable jobs after 45s and fail as AWOL only after awol_timeout (AWOL plan, phase 4) Nov 12, 2026
@prasadlohakpure
prasadlohakpure force-pushed the awol/phase-4-awol-timeout branch from 1662101 to 15b5548 Compare November 12, 2026 17:52
@prasadlohakpure
prasadlohakpure force-pushed the awol/phase-4-awol-timeout branch from 15b5548 to 20b0d33 Compare November 13, 2026 06:38

for _, jb := range jobs {
if _, ok := j.commandHandlers[jb.CommandID].(plugin.Resumer); ok && jb.Status == jobStatus.Running {
if _, ok := j.commandHandlers[jb.CommandID].(plugin.Resumer); ok && (jb.Status == jobStatus.Accepted || jb.Status == jobStatus.Running) && !awol[jb.SystemID] {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awol_timeout doesn't limit how long a resumable job can go unwatched
Severity: Medium

awol is computed from aj.last_heartbeat. Two things reset or pre-empt that value:

  • Any live janitor releases the job once it has been stale for stale_job (45 s), and active_jobs_release.sql sets last_heartbeat to null. A null heartbeat is never a candidate again (last_heartbeat > 0).
  • The next claim (active_select.sql) sets last_heartbeat = now().

So the job can only be failed as AWOL when no janitor runs for the whole hour, meaning every agent is down. A resumable job that keeps losing its agent never gets there. Examples: Resume OOMs or panics the agent, or the agent crash-loops on that job. Each time, the job is released after about 45 s, claimed again with a fresh heartbeat, and the cycle repeats forever. Every cycle can take down another agent. The PR describes this as "fail them as AWOL only after awol_timeout", but the code never applies that limit, and scenario 5 can only happen during a full-fleet outage.

Fix: measure awol_timeout from a timestamp that a claim doesn't reset. One option is a stale_since/first_released_at column on active_jobs. Set it on the first release, keep it through claims, clear it when the job finishes, and compute awol from coalesce(stale_since, last_heartbeat). Another option is a release counter with a maximum. Also add a test case to TestStaleJobs for a job that is released, claimed again and goes stale again.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. awol_timeout bounds a released job that no agent picks up; as of dae84ec, release keeps last_heartbeat, so that is measured from the last real heartbeat. A claim still sets last_heartbeat = now() (otherwise a just-claimed job would look stale before its first keepalive), so a job that crashes every agent that claims it keeps getting released and claimed. Bounding that needs a timestamp a claim does not reset, or a release counter; tracking it as a follow-up and noting it under Known gaps.

@prasadlohakpure
prasadlohakpure requested review from a team and Nitin Bhakar (nitinbhakar) and removed request for Nitin Bhakar (nitinbhakar) November 13, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants