feat(integrations): map reasoning effort and context size for Copilot, Codex, and Claude Code by sukanth · Pull Request #5249 · github/spec-kit

sukanth · 2026-11-05T08:27:59Z

Description

Closes #4916.

Builds on #5248, which adds the shared per-step dispatch-settings contract. That PR must merge first; until then, this branch also contains its two commits. Once it merges, only the five commits below remain. This split follows Integration pull requests.

This PR maps the workflow step settings reasoning_effort and context_size to the native CLI arguments of three integrations. It has one commit per integration, one docs commit, and one review fix.

Integration	`reasoning_effort`	`context_size`	Validation
GitHub Copilot	`--reasoning-effort <v>`	`--context <v>`	None. Values pass through, and Copilot CLI 1.0.106 rejects unknown values itself.
Codex CLI	`-c model_reasoning_effort="<v>"`	`-c model_context_window=<n>`	`context_size` must be a positive whole number with no leading zeros, at most `2**63 - 1`, the range Codex's i64 config accepts.
Claude Code	`--effort <v>`	Not mapped. The step fails with a hint to choose a long-context `model`.	`reasoning_effort` must be `low`, `medium`, `high`, `xhigh`, or `max`. Claude Code 2.1.318 only warns about other values and runs at its default effort.

Copilot:
- Both receiving methods declare and translate the settings: build_exec_args() for prompt steps, and the overridden dispatch_command() for command steps.
- One private helper builds the flags for both argv paths, in both the skills and commands layouts.
- The flags go after --model and before --output-format.
Codex: the -c overrides go after --model. A value is embedded as a TOML string only after it passes the host's value check, so it cannot contain quotes, backslashes, or control characters.
Claude Code:
- ClaudeIntegration now owns build_exec_args(). It calls super() without the settings, which validates the runtime configuration once and builds the base argv. It then runs the gate and appends --effort.
- With both settings unset, it returns super()'s argv unchanged.
- The error for an unknown level names an escape hatch for levels added in newer Claude Code releases: SPECKIT_INTEGRATION_CLAUDE_EXTRA_ARGS="--effort <value>".
Rejected values are not repeated. The Codex context_size error and the Claude Code reasoning_effort error state the rule but not the value. A value can come from captured agent output, which Spec Kit does not print to the terminal or write to log.jsonl.
Unset settings leave argv unchanged. With both settings unset, every integration builds byte-identical argv. Before editing, a 144-case argv snapshot was taken; afterwards the output was compared to it with cmp. The snapshot covers:
- all three integrations;
- build_exec_args and dispatch_command;
- Copilot's skills and commands layouts;
- --yolo on and off;
- EXTRA_ARGS set and unset;
- model set and unset;
- output_json, streaming, and bounded capture.
Docs:
- Each of the three agent guides gets a "Workflow Dispatch Settings" section.
- docs/workflow-steps/types.md gets a support table and the Claude example line.
- docs/integrations/cli.md lists the built-in support.

Testing

The agent described in the AI disclosure ran all of the testing below. Nothing was run by hand. The commands ran against this PR's head commit, 3d587126, in a clean worktree.

Tested locally with uv run specify --help: exit 0.
Ran existing tests with uv sync && uv run pytest, in an equivalent .venv. Results are below.
Tested with a sample project. See "Manual test results".

Command	Result
`uvx ruff@0.15.0 check src tests`	All checks passed
`npx markdownlint-cli2@0.23.2` on the eight changed docs	0 issues; all relative links and anchors resolve
`.venv/bin/python -m pytest tests/workflows tests/test_workflows.py tests/integrations tests/specify_cli/integrations tests/test_agent_config_consistency.py -q -n 6`	6298 passed, 8 skipped
New tests only (`-k DispatchSettings` in the three integration test files)	74 passed: 19 Copilot, 26 Codex, 29 Claude Code
`.venv/bin/python -m pytest -q -n 6` (full suite), then a serial re-run of every failure	Under `-n 6`: 14325 passed, 365 failed, 132 skipped. All 365 failures are in the set `main` fails under the same command: shell-script and parity suites that fail under parallel load on this machine. A serial re-run of those 365 passed 364. The remaining test is the environment-specific `test_non_windows_uv_tool_dry_run_still_previews_installer`, which also fails on `main` here.

Before and after. On e3bd6a0d, the new integration tests ran twice against the prerequisite PR's contract: once with the three integration modules from main, and once with this PR's modules. For the first run, the Claude module from main got only the EFFORT_LEVELS constant, so that the test module imports.

Before: 63 failed, 5 passed. The five pass by design: a Copilot ownership negative (a legacy builder subclass rejects the prompt path), the injected Claude vocabulary constant, Claude's unset-argv invariant, the order of Claude's runtime-configuration and setting errors, and the existing EXTRA_ARGS escape hatch.
After: 68 passed.

Review fix. The six tests added in 3d587126 feed captured output that passes the host's value check into the Codex context_size and the Claude Code reasoning_effort, both directly and through command and prompt steps. All six failed on the rebased cc6ce181, whose errors repeated the value, and pass on 3d587126.

Test selection reasoning

Changed file	Affects	Test	Why
`src/specify_cli/integrations/copilot/__init__.py`	`specify workflow run` command and prompt steps with Copilot	T1, T2	Copilot argv paths
`src/specify_cli/integrations/codex/__init__.py`, `claude/__init__.py`	`specify workflow run` with Codex or Claude Code	Unit tests, T3, and parser evidence	Neither CLI is authenticated on the test machine, so T3 uses a stand-in executable
`docs/**`	none	none	Documentation only

Manual test results

Agent: GitHub Copilot CLI 1.0.106, run by specify workflow run through an argv-logging probe that appends --version. The real CLI parses Spec Kit's argv without making a model call. Claude Code 2.1.318 and Codex CLI 0.165.0 were checked at the parser level only. OS/Shell: macOS / zsh

Command tested	Notes
T1 `specify workflow run` with three steps: command step `plan` (`reasoning_effort: "{{ inputs.effort }}"`, `context_size: long_context`); prompt step `review` (inherits `workflow.reasoning_effort: medium`); command step `tasks` (opts out with `reasoning_effort: "{{ inputs.tasks_effort }}"`, an input that defaults to `""`; a literal `""` would inherit the default)	Passed. The Copilot CLI received `--reasoning-effort low --context long_context`, then `--reasoning-effort medium`, then no new flags. All three steps completed with exit 0. The run state records each step's values: `low`/`long_context`, `medium`/`null`, and `null`/`null`.
T2 `specify workflow run` with `reasoning_effort: turbo` and `capture: true`	Passed. The Copilot CLI rejected the value, and the step failed with `Command exited with code 1`. The CLI's `invalid value 'turbo'` message is in `output.stderr`.
T3 `specify workflow run` in which a `capture: true` prompt step's output (`FAKEsecret0123456789`, printed by a stand-in executable) feeds the next step's `context_size` (Codex) or `reasoning_effort` (Claude Code)	Passed. Each run failed at the second step before the CLI started. The captured text is not in the step error, `log.jsonl`, or the console output. On `e3bd6a0d` it was in all three.
Claude Code `--effort` and Codex `-c` overrides	`claude -p … --effort bogus` only warns, which is why Spec Kit validates the level. `codex exec -c 'model_reasoning_effort="high"'` reports `reasoning effort: high`. A non-integer `model_context_window` is rejected (`expected i64`).

AI Disclosure

I did not use AI assistance for this contribution
I did use AI assistance (fill in the disclosure below)

AI disclosure: Implemented with the GitHub Copilot app in agent mode, running autonomously on behalf of @sukanth, who chose the issue and directed the work. An orchestrator and four implementation workers, all Claude Opus 5.5 at maximum reasoning effort with long context, wrote the change. Before implementation, a panel of five models reviewed and approved the plan: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol, and GPT-5.6 Sol at maximum reasoning effort, and Gemini 4 Argon at high reasoning effort, all with long context. The same panel reviewed the code, and the agent fixed the findings it accepted. Extent: AI generated the code, tests, docs, and this description, and the agent ran all of the testing. No human reviewed the change line by line before submission. AI-generated follow-up commits and review replies will carry their own disclosure. The first review fix (3d587126) and this description's re-run were done by the Agent merge session for this PR, also Claude Opus 5.5 at maximum reasoning effort, with no line-by-line human review.

Workflow command and prompt steps can set reasoning_effort and context_size, and workflow.reasoning_effort and workflow.context_size set defaults for both. Values resolve like model: the step value, then the workflow default, then unset. An expression that resolves to an empty string or null unsets the setting for that step. Values must be strings that match a host-owned, flag-safe pattern. Malformed values are summarized rather than echoed. An integration receives a setting only if it declares the setting on the class that implements the receiving method (build_exec_args, or dispatch_command when overridden) and translates it. Otherwise the step fails before the CLI runs, with an error that names the setting and its source. The base and format-base builders raise rather than drop a setting. With both settings unset, argv and integration keyword arguments are unchanged. Step outputs record both values. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Validation diagnostics can expose secret-bearing captured output in step errors and log.jsonl, violating the capture guarantee and requiring value-free diagnostics with regression coverage.

1 open issue

Redact secret-bearing values from dispatch-setting validation errors · New

What changed in this PR

Adds per-step reasoning effort and context size to Spec Kit workflow dispatch, including the shared contract from prerequisite #5248 and native mappings addressing #4916.

Changes:

Adds workflow defaults, expression resolution, validation, and run/resume recording for both settings.
Maps both settings for Copilot and Codex, and reasoning effort for Claude Code, while preserving arguments when unset.
Adds contract, adapter, and integration tests, plus documentation of support and limitations.

File	Description
`tests/workflows/test_dispatch_settings.py`	Tests resolution, validation, dispatch, and resume behavior.
`tests/specify_cli/integrations/test_installed_adapters.py`	Tests installed-adapter support and legacy compatibility.
`tests/integrations/test_integration_copilot.py`	Tests Copilot flags across both layouts and dispatch paths.
`tests/integrations/test_integration_codex.py`	Tests Codex overrides and token-count validation.
`tests/integrations/test_integration_claude.py`	Tests Claude effort levels and unsupported context settings.
`tests/integrations/test_base.py`	Tests shared support declarations and validation gates.
`src/specify_cli/workflows/step/prompt/__init__.py`	Resolves and forwards prompt-step settings.
`src/specify_cli/workflows/step/command/__init__.py`	Resolves and forwards command-step settings.
`src/specify_cli/workflows/step/_dispatch_settings.py`	Shares resolution, validation, and error attribution.
`src/specify_cli/workflows/engine.py`	Validates defaults and supplies them during run/resume.
`src/specify_cli/workflows/base.py`	Appends dispatch defaults to `StepContext`.
`src/specify_cli/workflows/_execution.py`	Gives called workflows their own dispatch defaults.
`src/specify_cli/integrations/copilot/__init__.py`	Maps settings to native Copilot flags.
`src/specify_cli/integrations/codex/__init__.py`	Maps TOML overrides and validates context counts.
`src/specify_cli/integrations/claude/__init__.py`	Maps effort and rejects unsupported settings.
`src/specify_cli/integrations/base.py`	Defines the shared opt-in dispatch contract.
`docs/workflows/api.md`	Documents workflow defaults and options semantics.
`docs/workflow-steps/types.md`	Explains settings, support, recording, and compatibility.
`docs/workflow-steps/api.md`	Documents new context fields.
`docs/integrations/copilot.md`	Documents Copilot flags and accepted values.
`docs/integrations/codex.md`	Documents Codex overrides and context semantics.
`docs/integrations/cli.md`	Summarizes shared dispatch support.
`docs/integrations/claude.md`	Documents effort validation and context limitations.
`design/integration.md`	Defines adapter ownership and opt-in requirements.

_{Review effort: Balanced (auto)}

A reasoning_effort or context_size value can come from an expression over captured agent output. When such a value failed the host's format check, the error quoted a 24-character repr prefix of it. Step errors are printed and written to log.jsonl as step_failed events, so captured text could reach both, contrary to the documented guarantee that captured text is neither printed nor written to log.jsonl. The error now reports only the value's type and length. The design doc and the validate_dispatch_setting() docstring tell adapter authors to keep the value out of their own errors for the same reason. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Translate the reasoning_effort and context_size workflow step settings to --reasoning-effort and --context on both Copilot argv paths: prompt steps through build_exec_args() and command steps through the overridden dispatch_command(). The flags follow --model. Copilot CLI validates the values itself. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Translate reasoning_effort to -c model_reasoning_effort="<value>" and context_size to -c model_context_window=<n> after --model. context_size must be a positive whole number no larger than the i64 range that Codex accepts. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Claude Code only warns about an unknown --effort value and then runs at its default effort, so Spec Kit checks the value against Claude's documented levels (low, medium, high, xhigh, max) before dispatch. The error names SPECKIT_INTEGRATION_CLAUDE_EXTRA_ARGS as an escape hatch for newer levels. context_size is not mapped; its error points to choosing a long-context model. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add the support table for GitHub Copilot, Codex CLI, and Claude Code to the workflow step reference and the shared CLI dispatch guide. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A dispatch setting can resolve from captured agent output, and a step error is printed and written to log.jsonl, where captured text must never appear. Claude Code's reasoning_effort check and Codex's context_size check repeated the rejected value, so captured text that passes the host's value pattern reached both. The errors now state the setting and its rule without the value. Part of github#4916. Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟢 Approval recommended

No active review issues were identified for the per-step reasoning-effort and context-size changes.

1 open issue

Redact secret-bearing values from dispatch-setting validation errors

_{Review effort: Balanced (auto)}

sukanth · 2026-11-05T09:17:39Z

Review round 1 (Copilot code review of e3bd6a0d)

This branch is rebased onto #5248's new head, 6a8ae513, and the head is now 3d587126. The rebase changed only the parents of the integration and docs commits.

Changes:

6a8ae513 in feat(workflows): add per-step reasoning effort and context size settings #5248, fix(workflows): keep rejected dispatch setting values out of errors: an error about a malformed dispatch-setting value reports only its type and length. design/integration.md now tells integrations to keep the value out of validate_dispatch_setting() errors.
3d587126, fix(integrations): omit rejected values from Claude and Codex errors: the Codex context_size error and the Claude Code reasoning_effort error state the rule without the value.

Evidence on 3d587126:

The six tests added in 3d587126 failed on the rebased cc6ce181 and pass now. feat(workflows): add per-step reasoning effort and context size settings #5248's description covers 6a8ae513.
T3 in the description is a specify workflow run in which captured output feeds the next step's setting. On e3bd6a0d, the captured text reached the step error, log.jsonl, and the console. On 3d587126, it reaches none of them.
ruff and markdownlint are clean. The focused suites passed, and the full suite has no failures outside main's own failure set. T1 and T2 were re-run with the same results. The description's Testing section has the details.

Posted on behalf of @sukanth by the GitHub Copilot app (agent mode, Agent merge, autonomous) with Claude Opus 5.5 at maximum reasoning effort. AI wrote the fix, the tests, and this summary, and ran all of the testing; no human reviewed them line by line.

mnriem · 2026-11-05T13:50:39Z

Please address Copilot feedback

sukanth · 2026-11-05T14:17:38Z

Please address Copilot feedback

The Copilot feedback on this PR is the one thread on base.py, and it is addressed. 6a8ae513 (in #5248) and 3d587126 keep rejected dispatch-setting values out of step errors, with regression tests, and the thread has a reply describing the fix. Copilot's re-review of 3d587126 reports no active issues, and CI passes on that commit. The author will resolve the thread; per CONTRIBUTING, the agent does not resolve reviewer conversations.

Posted on behalf of @sukanth by the GitHub Copilot app (agent mode, Agent merge, autonomous) with Claude Opus 5.5 at maximum reasoning effort. AI wrote this reply; no human reviewed it before posting.

mnriem · 2026-11-05T17:41:45Z

The Copilot feedback is addressed, and that review conversation is resolved. This PR remains blocked pending the merge of #5248; once that prerequisite merges, please rebase this branch to remove the prerequisite commits so the remaining changes can receive final review before merging.

AI disclosure: Posted on behalf of @mnriem by the GitHub Copilot app using GPT-6.1 Sol in interactive agent mode. AI drafted and posted this comment at the maintainer’s direction; model settings are not exposed to the agent.

sukanth requested a review from mnriem as a code owner November 5, 2026 08:28

Copilot AI auto review requested due to automatic review settings November 5, 2026 08:28

Copilot started reviewing on behalf of sukanth November 5, 2026 08:28

View session

sukanth mentioned this pull request Nov 5, 2026

feat(workflows): add per-step reasoning effort and context size settings #5248

Open

4 of 5 tasks

Copilot AI reviewed Nov 5, 2026

View reviewed changes

Comment thread src/specify_cli/integrations/base.py Resolved

Sukanth Gunda and others added 6 commits November 5, 2026 02:54

sukanth force-pushed the feat/4916-dispatch-settings-integrations branch from e3bd6a0 to 3d58712 Compare November 5, 2026 09:07

Copilot AI auto review requested due to automatic review settings November 5, 2026 09:07

Copilot started reviewing on behalf of sukanth November 5, 2026 09:07

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Nov 5, 2026

mnriem added the author-awaiting Waiting on author response label Nov 5, 2026

Navigation Menu

Repository navigation

feat(integrations): map reasoning effort and context size for Copilot, Codex, and Claude Code - #5249

feat(integrations): map reasoning effort and context size for Copilot, Codex, and Claude Code#5249
sukanth wants to merge 7 commits into
github:mainfrom
sukanth:feat/4916-dispatch-settings-integrations

sukanth commented Nov 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

sukanth commented Nov 5, 2026

Uh oh!

mnriem commented Nov 5, 2026

Uh oh!

sukanth commented Nov 5, 2026

Uh oh!

mnriem commented Nov 5, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sukanth commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Test selection reasoning

Manual test results

AI Disclosure

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

sukanth commented Nov 5, 2026

Uh oh!

mnriem commented Nov 5, 2026

Uh oh!

sukanth commented Nov 5, 2026

Uh oh!

mnriem commented Nov 5, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sukanth commented Nov 5, 2026 •

edited

Loading