Skip to content

test: a gh simulator with the read paths (#86) - #90

Merged
glennfaison merged 2 commits into
mainfrom
feat/gh-simulator-read-paths
Nov 10, 2026
Merged

glennfaison merged 2 commits into
mainfrom
feat/gh-simulator-read-paths

Conversation

@glennfaison

Copy link
Copy Markdown
Owner

Closes #86. Part 1 of 3 of #60 (next: #87 write paths, #88 api, faults, one mock).

What

A gh simulator for tests. tests/sim/bin/gh is an executable that a runner puts first on PATH. It answers gh calls from a world file in the scenarios' format (GH_SIM_WORLD) and appends every call to a log (GH_SIM_LOG).

  • Read paths: issue list|view, pr list|view|checks, label list, repo view, auth status, run view --log|--log-failed, with --json, --jq, --label, --state, --limit, --search and -R.
  • Loud failure: a command, flag or --json field it does not cover exits 1 with unsupported: <what> and is logged as unsupported. A field gh itself rejects fails with gh's own message.
  • Manifest: tests/sim/surface.mjs lists every gh command the skills use as supported, planned or out-of-scope. A test greps skills/ and scripts/ and fails for a command that is not listed. The supported entries must equal the simulator's handlers.
  • Seeding: node tests/sim/seed.mjs <scenario> <world> makes a world file a run can write to. A scenario file is read as it is (label catalog expanded). Every one of the 89 golden scenarios loads and reads (tested).
  • docs/adr/0014-gh-simulator.md, a rule in AGENTS.md, a changeset.

How it stays faithful to gh

I did not write gh's behaviour from memory. I read gh v2.63.2's source and compared with a real gh run against this repository (read only):

  • --json field lists come from api/query_builder.go (v2.80).
  • --jq prints as go-gh does: a string raw, null as an empty line, whole numbers plain.
  • Plain list output matches what a real gh printed to a pipe. Plain issue view and pr view follow printRawIssuePreview and printRawPrPreview. (I could not run those two live: this repository's classic-Projects deprecation makes gh issue view fail here.)
  • pr checks exits 1 on a failure and 8 on a pending check without --json, and always 0 with it (checks.go).

What it found

Following gh exactly shows two places where the package assumes something gh does not do:

  1. prs_for_issue and issue_for_pr ask gh for authorAssociation, which is not a PR field #89 (filed, not fixed here). lib.sh (prs_for_issue, issue_for_pr) asks gh pr list|view for authorAssociation. I confirmed against a real gh that it is not a PR field, and it is not in any release's list through v2.80 or trunk. Both functions fail on a real gh. The stub in tests/lib.test.sh ignores --json fields, which is why 154 tests never noticed. When gh simulator (3/3): api, fault injection, and one mock for lib.test.sh #88 replaces the stub with this simulator, those tests will fail until prs_for_issue and issue_for_pr ask gh for authorAssociation, which is not a PR field #89 is fixed, so prs_for_issue and issue_for_pr ask gh for authorAssociation, which is not a PR field #89 should land before gh simulator (3/3): api, fault injection, and one mock for lib.test.sh #88.
  2. The shepherd skill's check-state one-liner expects gh pr checks N --json bucket to print [] for a PR with no checks, giving no-checks-yet. A real gh exits 1 with no checks reported on the '<branch>' branch and prints nothing. The skill also says the exit code is non-zero while checks run, which is true only without --json. The simulator does what gh does, and tests/gh-sim.test.mjs records the difference. I have not changed the skill; say if you want that as its own issue.

Choices worth a look

  • State shown as comments. lease, readySha, openFindings and shepherdAttempts are world fields, but lib.sh reads <!-- agent:... --> markers from comments. The simulator shows them as comments (views, never stored back, so diffWorlds is unaffected). A stored comment that already carries the marker stands in for the derived one. A ready-for-review PR is shown with a ready handoff ten minutes old, so it holds the merge lane.
  • Not shown. entries, kind, openQuestions, useCase, wontfix stand for facts a model would read from prose. No gh call shows them, and seed.mjs prints which ones a world has. Scenarios that depend on them need a runner to write the prose (Run a model through a skill against the simulator and check the outcome #61).
  • Approximations, all in the ADR: --search is every-word-in-title-or-body plus closed:>=DATE (not GitHub's search); a PR is on feat/<closed issue>-pr-<N> unless it names a branch; a check is check-<i>; the log of a failing run is one fixed line.
  • Field lists are a copy of one gh version and can fall behind.

Checks

npm run check passes: lint, validate, plugin version, 154 shell tests (unchanged), 407 node tests (371 before, 36 new).

glennfaison and others added 2 commits November 9, 2026 02:01
Part of #60. tests/sim/bin/gh answers gh calls from a world file in the
scenarios' format and logs every call. It covers issue list|view,
pr list|view|checks, label list, repo view, auth status and run view,
checks --json fields against the ones gh has, and fails with
`unsupported: <what>` for anything else. A manifest and a test notice a
new gh call in a skill.

Co-authored-by: Claude <noreply@anthropic.com>
jq exits before it reads its input when it cannot compile the expression,
and spawnSync can report that as EPIPE. The simulator took any spawn error
for a missing jq, so a bad --jq expression sometimes exited 2 with the wrong
message. Only ENOENT means jq is missing; otherwise the exit status counts.

Co-authored-by: Claude <noreply@anthropic.com>
@glennfaison
glennfaison merged commit 302d78c into main Nov 10, 2026
2 checks passed
@glennfaison
glennfaison deleted the feat/gh-simulator-read-paths branch November 10, 2026 01:03
@glennfaison
glennfaison restored the feat/gh-simulator-read-paths branch November 10, 2026 01:04
@glennfaison
glennfaison deleted the feat/gh-simulator-read-paths branch November 10, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gh simulator (1/3): the world file, the call log and the read paths

1 participant