Repository navigation
test: a gh simulator with the read paths (#86) - #90
Merged
Merged
Conversation
Part of #60. tests/sim/bin/gh answers gh calls from a world file in the scenarios' format and logs every call. It covers issue list|view, pr list|view|checks, label list, repo view, auth status and run view, checks --json fields against the ones gh has, and fails with `unsupported: <what>` for anything else. A manifest and a test notice a new gh call in a skill. Co-authored-by: Claude <noreply@anthropic.com>
jq exits before it reads its input when it cannot compile the expression, and spawnSync can report that as EPIPE. The simulator took any spawn error for a missing jq, so a bad --jq expression sometimes exited 2 with the wrong message. Only ENOENT means jq is missing; otherwise the exit status counts. Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #86. Part 1 of 3 of #60 (next: #87 write paths, #88
api, faults, one mock).What
A
ghsimulator for tests.tests/sim/bin/ghis an executable that a runner puts first onPATH. It answersghcalls from a world file in the scenarios' format (GH_SIM_WORLD) and appends every call to a log (GH_SIM_LOG).issue list|view,pr list|view|checks,label list,repo view,auth status,run view --log|--log-failed, with--json,--jq,--label,--state,--limit,--searchand-R.--jsonfield it does not cover exits 1 withunsupported: <what>and is logged asunsupported. A fieldghitself rejects fails withgh's own message.tests/sim/surface.mjslists everyghcommand the skills use assupported,plannedorout-of-scope. A test grepsskills/andscripts/and fails for a command that is not listed. Thesupportedentries must equal the simulator's handlers.node tests/sim/seed.mjs <scenario> <world>makes a world file a run can write to. A scenario file is read as it is (label catalog expanded). Every one of the 89 golden scenarios loads and reads (tested).docs/adr/0014-gh-simulator.md, a rule inAGENTS.md, a changeset.How it stays faithful to
ghI did not write
gh's behaviour from memory. I readghv2.63.2's source and compared with a realghrun against this repository (read only):--jsonfield lists come fromapi/query_builder.go(v2.80).--jqprints asgo-ghdoes: a string raw,nullas an empty line, whole numbers plain.ghprinted to a pipe. Plainissue viewandpr viewfollowprintRawIssuePreviewandprintRawPrPreview. (I could not run those two live: this repository's classic-Projects deprecation makesgh issue viewfail here.)pr checksexits 1 on a failure and 8 on a pending check without--json, and always 0 with it (checks.go).What it found
Following
ghexactly shows two places where the package assumes somethingghdoes not do:prs_for_issueandissue_for_praskghforauthorAssociation, which is not a PR field #89 (filed, not fixed here).lib.sh(prs_for_issue,issue_for_pr) asksgh pr list|viewforauthorAssociation. I confirmed against a realghthat it is not a PR field, and it is not in any release's list through v2.80 or trunk. Both functions fail on a realgh. The stub intests/lib.test.shignores--jsonfields, which is why 154 tests never noticed. When gh simulator (3/3): api, fault injection, and one mock for lib.test.sh #88 replaces the stub with this simulator, those tests will fail untilprs_for_issueandissue_for_praskghforauthorAssociation, which is not a PR field #89 is fixed, soprs_for_issueandissue_for_praskghforauthorAssociation, which is not a PR field #89 should land before gh simulator (3/3): api, fault injection, and one mock for lib.test.sh #88.gh pr checks N --json bucketto print[]for a PR with no checks, givingno-checks-yet. A realghexits 1 withno checks reported on the '<branch>' branchand prints nothing. The skill also says the exit code is non-zero while checks run, which is true only without--json. The simulator does whatghdoes, andtests/gh-sim.test.mjsrecords the difference. I have not changed the skill; say if you want that as its own issue.Choices worth a look
lease,readySha,openFindingsandshepherdAttemptsare world fields, butlib.shreads<!-- agent:... -->markers from comments. The simulator shows them as comments (views, never stored back, sodiffWorldsis unaffected). A stored comment that already carries the marker stands in for the derived one. Aready-for-reviewPR is shown with a ready handoff ten minutes old, so it holds the merge lane.entries,kind,openQuestions,useCase,wontfixstand for facts a model would read from prose. Noghcall shows them, andseed.mjsprints which ones a world has. Scenarios that depend on them need a runner to write the prose (Run a model through a skill against the simulator and check the outcome #61).--searchis every-word-in-title-or-body plusclosed:>=DATE(not GitHub's search); a PR is onfeat/<closed issue>-pr-<N>unless it names abranch; a check ischeck-<i>; the log of a failing run is one fixed line.ghversion and can fall behind.Checks
npm run checkpasses: lint, validate, plugin version, 154 shell tests (unchanged), 407 node tests (371 before, 36 new).