Skip to content

test: the gh simulator writes (#87) - #92

Merged
glennfaison merged 1 commit into
mainfrom
feat/gh-simulator-write-paths
Nov 10, 2026
Merged

glennfaison merged 1 commit into
mainfrom
feat/gh-simulator-write-paths

Conversation

@glennfaison

Copy link
Copy Markdown
Owner

Closes #87. Part 2 of 3 of #60 (next: #88).

Stacked on #90. The base is feat/gh-simulator-read-paths, so this diff shows only the write paths. When #90 merges and its branch is deleted, GitHub retargets this PR to main.

What

The gh simulator now writes: issue edit|close|create|comment|reopen, pr edit|create|comment|merge and label create, with every flag the skills use (--add-label, --remove-label, --add-reviewer, --body-file from a path or stdin, --reason, --comment, --match-head-commit, --squash|--merge|--rebase, --delete-branch, --color, --description, --force, ...). tests/sim/surface.mjs moves these from planned to supported; only api is still planned.

  • A write is an effect. Each call is applied with applyEffects, the function the checker uses. So a run made of these calls is read back by diffWorlds as the effects it had, and applyEffects(before, diffWorlds(before, after)) gives the world the run left.
  • Check first, then write. A failing call exits non-zero and never writes. The file is written whole through a temporary file and a rename, in its own format (JSON stays JSON). A test asserts a long list of failing calls leave the file byte for byte unchanged.
  • Refused as gh refuses: a label outside the catalog ('x' not found), an empty comment, a missing --title/--body, a second open PR for the same branch, a merge with no method or two, a merge pinned to a head that is not the head, a merge of a draft or a DIRTY/BEHIND/BLOCKED PR, an invalid --reason. Merge messages follow gh's source (merge.go).
  • Never writes to a scenario file. Exit 2, with the seed.mjs command to run.
  • Written comments are fresh. Author is the agent, time is the call's clock, marker is the first <!-- agent:... --> in the body. A ready handoff the run just posted holds the merge lane.

A change to the checker's effect model

This touches scripts/scenario.mjs, so please look at it. A run that opens a PR gives it a title and a body, and diffWorlds reported both as unmodelled. That would make the checker reject every implement run unless its contract allowed an unmodelled write. So:

  • A create_issue / create_pr effect can carry data: title, body, createdAt, author, branch. Editing a title or body of an item that already existed is still unmodelled.
  • A close effect can carry a reason.

Both are optional, so no existing scenario changes. tests/diff-worlds.test.mjs has new tests for each, and the round trip still holds. ADR 0012 and 0014 are updated.

Choices worth a look

  • A merge does not close the issue. The skills settle the issue themselves after a merge (issue-after-pr.md), with a keep-open rule that GitHub's auto-close would hide, so the simulator leaves it open.
  • issue reopen is unmodelled. No effect op reopens an item, so a run that reopens one needs a contract that allows it.
  • pr create uses the checkout. Branch is --head or the current git branch, and the commit is that branch's, so it runs in a git repository. The issues it closes are the existing ones the body names with a closing keyword.
  • Approximate messages: the error for merging a closed PR and for an unreadable body file are close to gh's, not copied. The ADR says so.

Verified

  • claim and release in the real lib.sh run through the simulator (they use only edit and comment). A claim leaves a label and an agent:lease comment, a release removes the label, and a nested claim does nothing.
  • Every golden scenario's effects, made as gh calls, leave the world applyEffects says. 37 scenarios are replayed, 48 have no effects, and 4 are listed on purpose as not replayable (two open a PR, whose commit is a git fact, and two sync the label catalog), so a new scenario with an effect no gh call makes is noticed.
  • npm run check passes: 154 shell tests (unchanged), 430 node tests (407 before, 23 new). Six runs in a row.

A bug this found in #90

While writing the tests, a --jq test failed intermittently under the full suite (2 of 3 runs). When jq cannot compile an expression it exits before reading stdin, and spawnSync can report that as EPIPE. The simulator took any spawn error for "jq is missing", so a bad --jq expression sometimes exited 2 with the wrong message. #90's CI passed by luck. I fixed it on #90's branch as its own commit (946b655) and rebased this PR on it, and the test now runs the bad expression ten times.

@glennfaison
glennfaison deleted the branch feat/gh-simulator-read-paths November 10, 2026 01:03
@glennfaison glennfaison reopened this Nov 10, 2026
@glennfaison
glennfaison changed the base branch from feat/gh-simulator-read-paths to main November 10, 2026 01:04
Part of #60. issue edit|close|create|comment|reopen, pr edit|create|comment|merge
and label create change the world file through applyEffects, so diffWorlds reads
a run back as the effects it had. A call is checked before it writes and a failed
call leaves the file as it was. A create_* effect can carry the new item's title,
body, creation time, author and branch as data, and a close its reason.

Co-authored-by: Claude <noreply@anthropic.com>
@glennfaison
glennfaison force-pushed the feat/gh-simulator-write-paths branch from 7f71f2f to 8442440 Compare November 10, 2026 01:05
@glennfaison
glennfaison merged commit 862e759 into main Nov 10, 2026
2 checks passed
@glennfaison
glennfaison deleted the feat/gh-simulator-write-paths branch November 10, 2026 01:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gh simulator (2/3): the write paths

1 participant