[ParidelPooya]

Summary

The service resumes a suspended MicroVM on the next request when its idle policy sets autoResumeEnabled. So the MicroVM operations no longer resume a MicroVM themselves. The SDK never calls ResumeMicrovm, and the function's role no longer needs lambda:ResumeMicrovm.

This change builds on #910, now on main. It keeps the order of the state check from #910: after a sign that the MicroVM does not run, the state check runs first, and only the invocation reserve limits it. It keeps the one internal denial error, MicrovmPermissionDeniedError. Only the GetMicrovm case remains, so its message names lambda:GetMicrovm.

Twin PR: aws/aws-durable-execution-sdk-js#1182

Design note (CONTRIBUTING rule 3)

1. Problem

The MicroVM operations resume a suspended MicroVM themselves: GetMicrovm, ResumeMicrovm, polling until RUNNING, a recorded resume mode, and a "may be suspended" mark after suspend_after. This is a large share of the code and of the review rounds, and it needs lambda:ResumeMicrovm in the function role. The service can do the same job.

2. Cause

The SDK was built before IdlePolicy.autoResumeEnabled was known and tested. Without it, a suspended MicroVM's endpoint answers 502 MICROVM_CONNECT_FAILED forever, so the SDK had to resume.

3. What a live probe showed

A probe ran 16 MicroVMs in us-east-1. Every MicroVM was terminated afterwards.

Finding	Value
A request to a SUSPENDED MicroVM with autoResumeEnabled: true resumes it	Yes. The endpoint holds the request about 750 ms (720 to 1,335 ms, 12 samples), then the worker answers.
Window right after SuspendMicrovm	For 4.4 to 4.8 s, a request gets 502 MICROVM_CONNECT_FAILED in about 200 ms and does not resume the MicroVM. GetMicrovm already says SUSPENDED.
During the suspend hook	502 INTERNAL_ERROR, 502 MICROVM_CONNECT_FAILED, or no answer. A request with no answer reaches the worker after the next resume; the worker dedupes by callback ID.
autoResumeEnabled: false, or no idle policy	502 MICROVM_CONNECT_FAILED forever; stays SUSPENDED
Idle policy fields	All three required. maxIdleDurationSeconds >= 60, max 2,147,483,647.
Idle timer	Counts only inbound requests. It suspended a running job after 66 s.
Worker hooks	resume hook fires on every auto-resume, before the request reaches the worker. The worker's 503 refusal never reached a client after an auto-resume.

The current request retries (250 ms doubling to 4 s, 60 s window) reach 7.75 s after 5 retries, so they already cover the 5 s window. With a retry_window under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds.

4. Options

Option	Result
A. Keep the SDK resume logic	Works with or without the policy. Keeps the code, the permission and the review cost.
B. Simplify: rely on autoResumeEnabled, configured by the developer. Chosen.	Removes the resume code and lambda:ResumeMicrovm. A developer who suspends must set the policy.
C. The SDK sets the idle policy itself	Hides a service setting, and must pick maxIdleDurationSeconds; a wrong value suspends long jobs. The developer knows their job length.
D. B plus a check that reads idlePolicy from GetMicrovm and fails fast when auto-resume is off	A guard for a developer mistake. Rejected (no guards; the guideline is documented).

5. Chosen option

1. The SDK never calls ResumeMicrovm.
2. The state check from fix(extras): fail microvm() fast on a dead MicroVM #910 stays, only to find a dead MicroVM. After a 502 MICROVM_CONNECT_FAILED or a request without an answer, the request step makes one GetMicrovm call, at most once per 2 s. The call runs first, before the retry window and the backoff, and only the invocation reserve limits it. With less than 1 s left before the reserve, no call starts, and the attempt ends with the retryable MicrovmEndpointUnavailableError.
   • TERMINATING, TERMINATED, not found: fail at once with MicrovmNotRunningError.
   • RUNNING, PENDING, SUSPENDING, SUSPENDED: keep retrying. The service resumes the MicroVM on a request.
   • AccessDenied: fail at once with MicrovmPermissionDeniedError (fix(extras): fail microvm() fast on a dead MicroVM #910 behavior). Its message names lambda:GetMicrovm.
   • Other errors: one warning per step attempt, keep retrying.
3. suspend_after (the function calls SuspendMicrovm) and the worker's auto_suspend_on_idle stay. Both require the developer's idle policy with autoResumeEnabled: true.
4. No check that the policy is set. A developer who suspends without it gets 502 until the retries end: about 6 minutes with the default strategy, then MicrovmDeliveryError.
5. Docs (README, docstrings of suspend_after, auto_suspend_on_idle, idle_policy):
   • "To suspend a MicroVM, set idle_policy with "autoResumeEnabled": True. The SDK does not resume a MicroVM; the service resumes it on the next request."
   • "Set maxIdleDurationSeconds above maximumDurationInSeconds (the session timeout plus 5 minutes). The idle timer counts only inbound requests, so a shorter value suspends a running job."
   • "Set suspendedDurationSeconds to the session timeout or more." What happens to a job when that time ends is not verified.
   • "Without autoResumeEnabled, a job after a suspend fails after its retries with MicrovmDeliveryError."
   • "A job that arrives more than about 5 seconds after a suspend takes about 1 s longer."
   • "With a retry_window under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds."
   • Permissions: the function role no longer needs lambda:ResumeMicrovm. It needs lambda:GetMicrovm for HTTP delivery, and lambda:SuspendMicrovm for suspend_after.

6. What it removes

• ensure_running, its loop _wait_until_running, EnsureRunningOptions (lifecycle.py).
• The SUSPENDED/SUSPENDING branch and the ResumeMicrovm call in resume_if_suspended. It is renamed check_not_ended, and it takes the client, the MicroVM ID, and the remaining-time function.
• The ResumeMicrovm case of MicrovmPermissionDeniedError.
• The resume-wait cap from fix(extras): fail microvm() fast on a dead MicroVM #910 (min(max_wait_seconds, retry_window_seconds)). The check no longer waits, so no wait remains to cap.
• The recorded resume mode: resume_before_delivery in LaunchResult and resume_from_config (operation.py, session.py). LaunchResult.from_dict ignores a resumeBeforeDelivery key, so an old checkpoint still replays.
• The _may_be_suspended mark and _resume_mode of MicrovmSession.
• The recheck path of send_job (request.py), and the resume argument of deliver. SendJobOptions.suspend_check is renamed state_check, and it takes no argument.
• SendJobOptions.window_started_at. With no check before delivery, it equaled the start of send_job.
• ResumeMicrovm from MicrovmApiOperation and from microvm_not_found.
• lambda:ResumeMicrovm from the README permissions and the example docs.

Stays: the request retries, the dead-MicroVM check and its order, MicrovmPermissionDeniedError, suspend_after, worker auto_suspend_on_idle, the worker's suspend/resume hooks, its 503 refusal and 30 s grace (still needed for the worker's own SuspendMicrovm), and its callback-ID dedupe (now also needed for late delivery after a frozen request).

7. Replay and compatibility

• Operation names and subtypes do not change.
• An execution in flight that launched in resume mode replays its launch checkpoint with resumeBeforeDelivery: true. The new code ignores the field. Its next job no longer resumes the MicroVM. It works only if that MicroVM was launched with autoResumeEnabled: true. A suspend_after session without an idle policy is affected too: its next job after the suspend no longer resumes the MicroVM and fails after its retries with MicrovmDeliveryError. The README says both. The package is unpublished.
• The two suspend examples and their cloud tests set the idle policy.

Source lines (aws-durable-execution-sdk-python-extras/src only)

	Lines
Removed	635
Added	245
Net	-390

Coverage counts 967 statements and 188 branches, down from 1,094 and 228 on main.

Tests

Check	Result
Extras unit tests, coverage	436 passed, 100% statements and branches (main: 490 passed, 100%)
Worker unit tests	473 passed
Examples tests	204 passed, 6 skipped (cloud-only)
mypy, extras src and tests, worker src and tests	clean
hatch fmt --check	no findings in the changed files; the linter and the formatter pass on every changed .py file
Cloud tests	run by CI

New and changed tests:

Test	What it proves
conftest.py fake resume_microvm raises, and the autouse fixture no_resume_microvm fails any test whose fake client recorded the call	No code path calls ResumeMicrovm
lifecycle_test.py::test_check_returns_the_endpoint_of_a_microvm_that_has_not_ended[RUNNING, PENDING, SUSPENDING, SUSPENDED]	One GetMicrovm call, no error, no resume
lifecycle_test.py::test_check_raises_the_state_error_for_a_microvm_that_ends[TERMINATING, TERMINATED], ..._for_a_removed_microvm	Fail fast
lifecycle_test.py::test_check_fails_at_once_when_get_microvm_is_denied	MicrovmPermissionDeniedError, message names lambda:GetMicrovm
operation_test.py::test_checks_the_state_after_a_sign_that_uses_up_the_retry_window[connect-failed, no-answer] (from #910)	The check runs first after a sign, so a 1-second window still gets it
operation_test.py::test_ends_an_attempt_of_a_suspended_microvm_without_a_deadline	Replaces the resume-wait cap test of #910. Without a Lambda deadline, a SUSPENDED MicroVM whose endpoint keeps answering 502 gets one GetMicrovm call, no ResumeMicrovm, and the window ends the attempt
operation_test.py::test_a_custom_strategy_sees_the_denial_error_type	A custom strategy sees MicrovmPermissionDeniedError for a denied GetMicrovm
request_test.py::test_runs_the_state_check_when_the_sign_used_up_the_window	With a 0-second window, the check still runs before the window ends the tier
request_test.py::test_starts_no_state_check_within_1_second_of_the_reserve	The reserve still applies: no GetMicrovm call
operation_test.py::test_keeps_sending_the_job_to_a_microvm_suspended_from_outside[SUSPENDING, SUSPENDED]	microvm keeps retrying
session_test.py::test_keeps_sending_to_a_microvm_that_the_worker_suspended_and_uses_its_endpoint[...], test_keeps_sending_the_job_after_a_suspend_after_job[...], test_a_job_after_a_replayed_suspend_after_job_keeps_sending, test_runs_both_items_when_both_find_the_microvm_suspended	Session jobs keep retrying after either suspend, also after replay and in a map
session_test.py::test_replays_a_launch_checkpoint_with_the_resume_mode_of_an_older_release	A launch checkpoint with resumeBeforeDelivery: true replays, and the next job works
operation_test.py::test_launch_result_ignores_the_resume_mode_of_an_older_release	from_dict ignores the field, to_dict does not write it
session_test.py::test_writes_no_resume_mode_in_the_launch_checkpoint[...]	New launches write no resume mode
worker_composed_test.py (3 tests)	With the real worker, a stand-in service answers the first request after a suspend with 502, then auto-resumes on the next request with the resume hook. Jobs succeed without ResumeMicrovm, with and without the resume hook, and when a resend meets a suspend.

Removed tests. The behaviors they covered no longer exist:

• Every ensure_running test, and the resume, resume-conflict and resume-wait tests of the state check (lifecycle_test.py).
• From fix(extras): fail microvm() fast on a dead MicroVM #910: lifecycle_test.py::test_check_fails_at_once_when_resume_microvm_is_denied, lifecycle_test.py::test_check_names_no_request_id_for_an_error_of_another_call (the ResumeMicrovm case; the GetMicrovm case stays), operation_test.py::test_the_wait_for_a_resume_gets_a_new_retry_window_without_a_deadline (replaced, see above), and the resume case of operation_test.py::test_a_custom_strategy_sees_one_error_type_for_a_denial.
• request_test.py::test_counts_the_window_from_the_given_start, with window_started_at.
• The resume-mode recording and deploy tests, the recheck tests of send_job, the "flag" tests of suspend_after, and the ResumeMicrovm scenarios of request_ids_test.py.

Edited expectations (rule 5.4):

• Tests that counted a GetMicrovm call before delivery now expect none, because no check runs before delivery.
• Tests that expected a dead MicroVM to fail before any request now send one request that gets 502 MICROVM_CONNECT_FAILED first, because the check runs only after that sign.
• request_ids_test.py: "GetMicrovm denies access" expects MicrovmPermissionDeniedError with its message, because the strict pre-check that raised the raw AccessDeniedException is gone. The message-based retry strategy case for GetMicrovm is removed, because a throttled state check logs a warning and never reaches the strategy.
• request_test.py::test_checks_the_state_after_a_connect_failure_or_no_answer: the check takes no argument, so the test records when it runs (0 and 3.75 s, before the delay) instead of the time it received.
• The warning text changes from "check whether the MicroVM is suspended" to "check whether the MicroVM has ended". The SuspendMicrovm warning no longer says "The next job checks its state."
• The unavailable message of a stalled GetMicrovm changes from "did not reach RUNNING" to "did not end before the invocation reserve", because the check no longer waits for RUNNING.

Mutation checks (rule 4)

Each row changes one branch in src, runs the extras tests, and restores the file.

Mutation	Result
M1 SUSPENDED/SUSPENDING fail fast	killed
M2 the state check calls ResumeMicrovm for SUSPENDED	killed (fake raises, autouse fixture fails)
M3 the state check runs after the window exit	killed
M4 a denied GetMicrovm is treated as transient	killed
M5 TERMINATING/TERMINATED no longer fail fast	killed
M6 not found no longer fails fast	killed
M7 the window limits the state check	killed
M8 no reserve guard before the state check	killed
M9 the warning repeats on each check	killed
M10 no state check after MICROVM_CONNECT_FAILED	killed
M11 no state check after a request without an answer	killed
M12 no 2 s spacing between checks	killed
M13 the endpoint that the check returns is ignored	killed
M14 from_dict rejects a resumeBeforeDelivery key	killed
M15 to_dict writes resumeBeforeDelivery again	killed

Assumptions (rule 6)

• Auto-resume, the 4.4 to 4.8 s window after SuspendMicrovm, the about 750 ms resume latency, and the idle timer that counts only inbound requests: verified by the live probe above.
• Auto-resume after the worker's own SuspendMicrovm call from inside the MicroVM: unverified. It is the same API call, so the same behavior is expected.
• About 6 minutes until MicrovmDeliveryError without autoResumeEnabled: computed from the default retry strategy, not measured.

Decisions not covered

• Whether outbound traffic, such as heartbeats, resets the idle timer. Not needed when maxIdleDurationSeconds is above the maximum duration.
• The behavior of suspendedDurationSeconds: 0.
• Whether to keep auto_suspend_on_idle at all. The service's idle timer with a long job is unsafe, and the worker's own suspend needs the same policy. It stays for now.
• The lambda:ResumeMicrovm grant in the role that the cloud tests use is defined outside this repository. Remove it in a later change, after both SDKs ship.
• Documentation-only changes in the worker: the docstrings and comments of worker.py, the docstring of SUSPEND_GRACE_SECONDS in self_suspend.py, two test comments, and the worker README now say that the durable function delivers the job again and the service resumes the MicroVM. The worker code is unchanged.
• SendJobOptions.suspend_check is renamed state_check, and resume_if_suspended is renamed check_not_ended with a new signature (client, MicroVM ID, remaining-time function).
• The message of a state check that reaches the invocation reserve is now GetMicrovm for MicroVM <id> did not end before the invocation reserve, after <n> ms. The JS twin uses the same message.
• No live run of this change outside CI. CI runs the cloud examples.

[github-actions]

Codex AI review

No actionable findings, including against CONTRIBUTING rules 1–9. Tests were not executed during this read-only review. Residual risk: the PR explicitly marks auto-resume after worker-initiated suspension as unverified.

Reviewed commit 3f441de62c5ba65ec7ebf33d91de7bf96e180da9. Workflow run