Repository navigation
feat: release resumable jobs after 45s and fail as AWOL only after awol_timeout (AWOL plan, phase 4) - #164
feat: release resumable jobs after 45s and fail as AWOL only after awol_timeout (AWOL plan, phase 4)#164prasadlohakpure wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
A claim/janitor race can incorrectly fail and clean up a newly reclaimed job, while cleanup calls without deadlines can delay release commits long enough to trigger duplicate submissions.
2 open issues
What changed in this PR
This implements phase 4 of the rollout recovery plan, separating prompt release of resumable jobs from the longer timeout for failing jobs as AWOL.
Changes:
- Sets the default resumable-job release threshold to 120 seconds and adds a separate 3600-second
awol_timeout. - Moves release-versus-fail decisions into SQL and preserves released jobs’ heartbeats so unclaimed work can eventually time out.
- Replaces job-splitting tests with coverage for identifying resumable commands.
| File | Description |
|---|---|
internal/pkg/janitor/queries/stale_and_canceling_jobs_select.sql |
Selects actionable jobs and determines whether to release or finalize them. |
internal/pkg/janitor/queries/active_jobs_release.sql |
Clears ownership while retaining the heartbeat for AWOL detection. |
internal/pkg/janitor/job.go |
Uses query-provided release decisions instead of splitting jobs in Go. |
internal/pkg/janitor/job_test.go |
Removes tests for the deleted splitting helper. |
internal/pkg/janitor/janitor.go |
Adds AWOL configuration and collects resumable command IDs. |
internal/pkg/janitor/janitor_test.go |
Tests resumable-command filtering and ordering. |
cmd/heimdall/heimdall.go |
Configures separate release and AWOL timeout defaults. |
Review effort: Balanced
d968a28 to
1662101
Compare
1662101 to
15b5548
Compare
…ol_timeout (AWOL plan, phase 4)
15b5548 to
20b0d33
Compare
|
|
||
| for _, jb := range jobs { | ||
| if _, ok := j.commandHandlers[jb.CommandID].(plugin.Resumer); ok && jb.Status == jobStatus.Running { | ||
| if _, ok := j.commandHandlers[jb.CommandID].(plugin.Resumer); ok && (jb.Status == jobStatus.Accepted || jb.Status == jobStatus.Running) && !awol[jb.SystemID] { |
There was a problem hiding this comment.
awol_timeout doesn't limit how long a resumable job can go unwatched
Severity: Medium
awol is computed from aj.last_heartbeat. Two things reset or pre-empt that value:
- Any live janitor releases the job once it has been stale for
stale_job(45 s), andactive_jobs_release.sqlsetslast_heartbeatto null. A null heartbeat is never a candidate again (last_heartbeat > 0). - The next claim (
active_select.sql) setslast_heartbeat = now().
So the job can only be failed as AWOL when no janitor runs for the whole hour, meaning every agent is down. A resumable job that keeps losing its agent never gets there. Examples: Resume OOMs or panics the agent, or the agent crash-loops on that job. Each time, the job is released after about 45 s, claimed again with a fresh heartbeat, and the cycle repeats forever. Every cycle can take down another agent. The PR describes this as "fail them as AWOL only after awol_timeout", but the code never applies that limit, and scenario 5 can only happen during a full-fleet outage.
Fix: measure awol_timeout from a timestamp that a claim doesn't reset. One option is a stale_since/first_released_at column on active_jobs. Set it on the first release, keep it through claims, clear it when the job finishes, and compute awol from coalesce(stale_since, last_heartbeat). Another option is a release counter with a maximum. Also add a test case to TestStaleJobs for a job that is released, claimed again and goes stale again.
There was a problem hiding this comment.
Agreed. awol_timeout bounds a released job that no agent picks up; as of dae84ec, release keeps last_heartbeat, so that is measured from the last real heartbeat. A claim still sets last_heartbeat = now() (otherwise a just-claimed job would look stale before its first keepalive), so a job that crashes every agent that claims it keeps getting released and claimed. Bounding that needs a timestamp a claim does not reset, or a release counter; tracking it as a follow-up and noting it under Known gaps.
…le rows against concurrent claims


AWOL-on-rollout plan
This PR is Phase 4. Phases 1 and 2 merged in #161, Phase 3 in #162.
plugin.Resumerforsparkeksandecs)awol_timeout; other jobs are failed after 45 s as before feat: Resume sparkeks and ecs jobs on another agent instead of failing them as AWOL (AWOL plan, phase 3) #162Problem
#162 raised
stale_jobto 1 hour so a short database or network hiccup wouldn't fail every non-resumable job. That one setting also decides when a resumable job is released, and an hour is too late for both resumable plugins:sparkeks: the SparkApplication template setstimeToLiveSeconds: 300. If the application finishes while no agent is watching, the operator deletes it after 5 minutes, and the resume an hour later submits it again.ecs: ECS keeps stopped tasks for about an hour. A task that stopped early in that hour may be gone by the time the job is released, and the resume fails.Job states and the
active_jobsrowStatuses (
job_statuses):new1,accepted2,running3,failed4,killed5,succeeded6,canceling7,canceled8. Nothing setskilledtoday.An
active_jobsrow is the lease for an async job:agent_namesays which agent holds it,last_heartbeatwhen that agent last reported in. Sync jobs never get a row; they run inside the request and are recorded injobsafterwards, so the janitor never sees them.jobs.job_status_idactive_jobsrownewagent_nameandlast_heartbeatnullinsertJob)newtoacceptedagent_nameset,last_heartbeatnowactive_select.sql,getAsyncJobs)acceptedtorunninglast_heartbeatrefreshed everykeepalive(5 s) while the job runsrunAsyncJob,jobKeepalive)succeededorfailedupdateAsyncJobStatus)failed,killed,succeeded,canceled)cancelingstatus_cancel_update.sql)cancelingcancelingrunJobcancel monitor, thenupdateAsyncJobStatus)cancelingtocanceledCleanupacceptedorrunningtofailedCleanupstale_job, changed here from 1 h to 45 s;acceptedjobs now too)acceptedorrunning)agent_nameset to null;last_heartbeatkeptactive_jobs_release.sql, only if the heartbeat is still older thanstale_job)runningstaysrunning;acceptedstaysacceptedagent_nameset,last_heartbeatnowrunJobcallsResumefor arunningjob,Executefor anacceptedoneA
cancelingjob is never claimed (active_select.sqlskips status 7), so a job canceled before any agent picked it up is canceled by the janitor.Behaviour per scenario
Times are without a heartbeat. "Resumable" means the job's command plugin implements
plugin.Resumer(sparkeks,ecs).running, resumable, stale for overstale_job(45 s)Resumeaccepted, resumable, stale for overstale_jobExecuteacceptedorrunning, resumable, heartbeat freshawol_timeoutagoacceptedorrunning, resumable, claimed or released, stale for overawol_timeout(1 h)Cleanupcalled. The janitor locks theactive_jobsrows, so a job an agent claims at that moment is left outacceptedorrunning, not resumable, stale for overstale_jobCleanupcalled (as before #162)acceptedorrunning, not resumable, heartbeat freshcanceling, noactive_jobsrow, or releasedCleanupcalled (as before)canceling, claimed, stale for overstale_jobCleanupcalled (as before #162)canceling, claimed, heartbeat freshnew, never claimedA job that finishes, fails or is canceled through its agent is never selected, since the agent removes its
active_jobsrow.Timing
From an agent dying, a resumable job is released within
stale_jobplus up toclean_interval(45 s + 60 s = under 2 minutes), and claimed by another agent within a second after that. That is inside Spark's 5-minute TTL and well inside ECS's stopped-task retention, so the resume finds the remote work.Testing
A. ECS Job started:
B. Heimdall rolls out
C. Job being stale for over 45 seconds

D. Job picked up
A. ECS Job started:
B. Heimdall rolls out
C. Job being stale for over 1 hour
D. Job AWOLed up
A. Shell plugin Job started:
B. Heimdall rolls out
C. Job being stale for over 45 seconds
D. Job AWOLed up
Known gaps
Agents are only replaced during an ECS rollout. A draining agent keeps heartbeating the jobs it holds, and async jobs run on
context.Background(), so at the drain deadline the process exits without writing a status. The job is then released 45 s later and resumed by a new agent. That rules out two agents watching one job, unless a database blip over 45 s cuts an agent's heartbeat while it is still running. What is still not covered:last_heartbeatto now, soawol_timeoutonly bounds a released job no agent picks up. Bounding crash loops needs a timestamp a claim doesn't reset, or a release counter; follow-up.timeoutrestarts on resume: polling measurestimeoutfrom the resume, so each rollout gives a running ECS job a fresh timeout.timeToLiveSeconds: 300in heimdall-config), and a later resume reruns the query. A release takes under 2 minutes, so this needs no agent up for over 5 minutes.