Skip to content

Record the CLI's commands and requests as golden files - #30

Merged
edumansky merged 3 commits into
mainfrom
golden-files
Nov 17, 2026
Merged

edumansky merged 3 commits into
mainfrom
golden-files

Conversation

@edumansky

@edumansky edumansky commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The CLI's published surface is now recorded in golden files, and a test fails if any of it changes.

This is PR 1 of the SDK extraction plan. PR 2 will move the CLI's API logic into pkg/. These files will show that the move leaves the CLI the same.

What's recorded

testdata/golden/:

  • commands.json: every command's path, aliases, help and flags, with each flag's type, default, usage, and whether it's required or hidden.
  • requests-<area>.json, for each of the six API areas: the exact requests each command sends, made through a recording transport with no network. Each request has its method, URL, headers and body.

Each command is run with:

  • every flag;
  • required flags only;
  • --body, if it takes one;
  • --paginate, if it sends a GET.

The fake server answers like a two-page list:

  • the first page is full, of the requested page size;
  • later pages are short;
  • the totals match;
  • a list that expects a JSON array gets one.

It also has download links, the agent list AI Agent Studio commands look up, a default greeting purpose, and Contact Center search pages.

The sample values satisfy the upload and search commands, and searches run with a time window and with --paginate. Multipart boundaries are fixed. So uploads, downloads with their file GET, searches and later pages are all recorded.

A command that shares its path with another one is never run by Cobra, so it is recorded as unreachable. A run that still fails is recorded with its error, which is the CLI's real response to those flags.

commands.json also records each command's use line, --version, and which flags are persistent.

Changes

  • cli_golden_test.go:
    • TestGoldenCommandTree and TestGoldenRequests.
    • They run with a temporary HOME and USERPROFILE, the file token store so the OS keychain is never read, cleared WEBEX_* variables, and a fake token and organization.
    • They pass alone and repeated.
    • TestGoldenFilesExerciseEachKindOfRequest checks that the files really contain the uploads, downloads (with no token on the file's URL), searches with their pages, later list pages and unreachable commands.
    • TestGoldenCommandTreeHasPublishedRootFlags checks --version and the persistent root flags.
  • .gitattributes keeps the golden JSON LF on every checkout.
  • make golden rewrites the files after an intended change.
  • The weekly collection refresh runs make golden, so its PR shows each command that changed.
  • AGENTS.md describes the files and when to update them.

This is #20's 36ef6fa, adapted to main. It has main's environment variables, the file token store, and the AI Agent Studio agent list in its paged form.

Checks

  • Coverage:
    • The files cover 2,346 commands and 6,196 runs. 6,185 of the runs send at least one request, and 318 --paginate runs read a later page.
    • 5 commands are unreachable.
    • 6 runs record an error, each the CLI's real response.
  • Stable: the tests pass alone, with -count=2, and in other timezones, and make golden changes nothing.
  • Catches a change: setting IdentityBaseURL to https://webexapis.com/v1 made the test fail on the archive-users URL, naming the line.
  • No real data: the files contain no real identity or tenant data. Every request carries the fake golden-token.
  • Test suites: go vet ./..., go test -p 1 ./... and the 74 codegen tests pass.

🤖 Generated with Claude Code

Every command's name, aliases, help and flags, and the exact request each API
command sends for a fixed set of flags (all flags, required flags only, --body,
--paginate), answered by a recording transport with no network. A change that
is meant to keep the CLI the same must leave these files unchanged.

make golden rewrites them, and the weekly collection refresh runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@edumansky edumansky left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex review (gpt-6.2-astra, reasoning effort: high)

The golden suite passes in its default order, but isolated and repeated runs fail on an unchanged checkout. Several request snapshots also pin fixture errors instead of the HTTP behavior they are intended to protect.

Comment thread cli_golden_test.go
var walk func(c *cobra.Command)
walk = func(c *cobra.Command) {
gc := goldenCommand{Path: c.CommandPath(), Aliases: c.Aliases, Short: c.Short, Long: c.Long, Hidden: c.Hidden}
c.LocalFlags().VisitAll(func(f *pflag.Flag) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Make the golden tests independent of Cobra execution order

go test -run '^TestGoldenCommandTree$' . fails on an unchanged checkout: Cobra adds --help lazily, so the committed tree includes flags installed by TestGoldenRequests. Conversely, go test -run Golden -count=2 . fails because the second request sweep passes --help=true and prints help instead of sending requests. Initialize the command-tree flags deterministically and exclude help from request arguments so the golden check works independently and on repeated runs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. The command tree now calls InitDefaultHelpFlag() on each command before recording it, so it no longer depends on which commands ran first. The request sweep adds help to skippedFlags, so a second sweep no longer passes --help=true. These now pass:

  • go test -run '^TestGoldenCommandTree$' .
  • go test -run '^TestGoldenRequests$' .
  • go test -run Golden -count=2 .

Comment thread cli_golden_test.go
case "duration":
return "90s"
default:
return "s-" + f.Name

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Record multipart uploads instead of missing-file errors

All 17 Calling and CC upload commands receive --file=s-file, which does not exist in the temporary directory. Every variant therefore records zero requests; changing an upload's method, route, or multipart field names would leave the goldens green. The recorded OS error also differs on Windows. Create a small fixture and record its multipart fields with the random boundary normalized so these commands meet the request snapshot contract.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. Each run's working directory now holds golden.wav, which is the sample --file. The recorder replaces the multipart boundary with golden-boundary in the header and the body. All 17 upload commands now record their multipart request. The new TestGoldenFilesExerciseEachKindOfRequest fails if an upload sends none; it failed on the previous golden files.

Comment thread cli_golden_test.go
required = append(required, arg)
}
})
runs := []goldenRun{run(rec, c, "all-flags", all), run(rec, c, "required", required)}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exercise the flag-built Contact Center search requests

For the six convenience searches, the all-flags run stops at invalid JSON in --filter=s-filter, and the required-only run stops because neither --from nor --to is marked required. The only recorded POST uses --body, which bypasses query construction entirely. Supply a valid fixed time window, filter, and aggregation for these commands so a change to the GraphQL request built from their published flags is visible in the request snapshots.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. The sample values are now --filter={"golden":true}, --aggregate=s-field:count:s-name, --interval=DAILY and --time-comparator=createdTime. With them, the all-flags run of each of the six searches records its flag-built GraphQL POST, and the regression test checks this. The required-only run still records --from and --to are required, because that is the command's real behaviour without those flags.

Comment thread cli_golden_test.go
// AI Agent Studio commands look up the agent's runtime token first.
if strings.HasSuffix(req.URL.Path, "/agents/") {
body = `{"objects":[{"id":"s-agent-id","bot_access_token":"golden-bot-token"}],"meta":{"total_count":1}}`
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Record the second request of binary downloads

The {} metadata response makes the CC audio/greeting and Calling/Meetings recording downloads return before fetching the binary. Their snapshots would still pass if the download GET sent the Webex bearer token to a pre-signed URL, contrary to the binary download invariant. Return a synthetic download URL in the expected metadata field and record the follow-up GET, including its headers.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. The fake response now carries url and temporaryDirectDownloadLinks, all on https://download.golden.example/. Each of the 4 download commands now records the follow-up GET with its headers. The regression test fails if that GET is missing, or if it carries an Authorization header.

Comment thread cli_golden_test.go
return &http.Response{
StatusCode: 200,
Header: http.Header{"Content-Type": {"application/json"}},
Body: io.NopCloser(strings.NewReader(body)),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exercise continuation requests in pagination snapshots

The recorder never supplies a second page, so the paginate snapshots cannot detect changes that affect only continuation requests. For example, removing LinkQueryOnly from admin resource-memberships list-group-v2 would change the URL used for page two without changing its golden. Use bounded two-page fixtures for the supported paging behaviors, including a next link whose path differs for this override.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. The fake response now serves two pages:

  • one item under each item field the lists read;
  • totals of two items or two pages;
  • on the first request only, a Link to /golden-next?golden_page=2, a different path.

296 --paginate runs now record a second page. The regression test also checks that admin resource-memberships list-group-v2 keeps its own path on page two, which is its LinkQueryOnly behaviour. Every run is capped at 20 requests, so a list can't loop forever; the most any run makes is 4.

Comment thread cli_golden_test.go
t.Helper()
home := t.TempDir()
t.Setenv("HOME", home)
t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config"))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Isolate the Windows user profile during golden runs

On Windows, appconfig.Dir() uses os.UserHomeDir(), which reads USERPROFILE, not HOME. These tests consequently load the developer's real Webex config, so a saved region can change the recorded hosts; aiagent.PrivateAPINotice() can also write its marker into the real profile. Set USERPROFILE to the temporary home as the existing cmd/ccregion_test.go isolation helper does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. goldenEnv now sets USERPROFILE to the temporary home as well as HOME, as cmd/ccregion_test.go does.

Comment thread cli_golden_test.go Outdated
var out []goldenCommand
var walk func(c *cobra.Command)
walk = func(c *cobra.Command) {
gc := goldenCommand{Path: c.CommandPath(), Aliases: c.Aliases, Short: c.Short, Long: c.Long, Hidden: c.Hidden}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Pin positional argument syntax in the command tree

CommandPath() includes only the command names, not the argument syntax in Cobra's Use field. For example, changing config set <key> <value> to config set <value> <key> would change the published usage without changing either golden test. Include Use or the usage line in the snapshot to cover the published help contract.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. Each command in commands.json now records use, cobra's Use line with its argument syntax.

Comment thread cli_golden_test.go
if err != nil {
t.Fatalf("%v (run make golden to create it)", err)
}
if !bytes.Equal(want, buf.Bytes()) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Keep golden comparisons stable on CRLF checkouts

With core.autocrlf=true, Git checks out these JSON files with CRLF, while json.Encoder emits LF on every platform. The byte comparison then rejects every snapshot even when the CLI is unchanged. Pin testdata/golden/*.json to LF in .gitattributes or normalize line endings before comparison.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8693db5. The new .gitattributes pins testdata/golden/*.json to eol=lf, and git check-attr confirms it applies.

The recorded runs stopped at fixture errors: a missing upload file, a search
filter that is not JSON, download metadata with no link, and lists with one page.
A shared fake response now carries a second page, download links and the agent
list, and the sample values satisfy the search and upload commands. Multipart
boundaries are fixed so uploads record stably.

The command tree records each command's usage and adds --help itself, and the
request runs leave --help out, so the tests pass alone and repeated. Windows'
USERPROFILE is isolated like HOME, and the golden JSON stays LF on checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@edumansky

Copy link
Copy Markdown
Contributor Author

Codex review: outcome

Codex (gpt-6.2-astra, reasoning effort: high) reviewed 0b46093 and made eight points. I checked each one against the code and confirmed it was real. All eight are fixed in 8693db5.

# Finding Outcome
1 [P2] Golden tests depend on Cobra execution order Fixed. The tree adds --help to each command itself, and the request runs leave --help out. They pass alone and with -count=2.
2 [P2] Uploads record missing-file errors Fixed. A golden.wav fixture is used, and the multipart boundary is fixed. All 17 uploads record a multipart request.
3 [P2] Contact Center searches never build their query Fixed. Valid --filter, --aggregate, --interval and --time-comparator samples mean all 6 searches record their GraphQL POST.
4 [P2] Downloads never fetch the binary Fixed. The fake metadata has download links. All 4 downloads record the follow-up GET, which is checked for no Authorization.
5 [P2] Paging never reads a second page Fixed. A shared two-page fake response and a first-page Link to a different path mean 296 --paginate runs read page two, and the LinkQueryOnly case is checked.
6 [P2] Windows USERPROFILE not isolated Fixed. It is set to the temporary home.
7 [P2] Argument syntax not recorded Fixed. commands.json records each command's use.
8 [P3] CRLF checkouts break the comparison Fixed. .gitattributes pins the golden JSON to LF.

Regression test: TestGoldenFilesExerciseEachKindOfRequest reads the golden files and checks the uploads, downloads (including no token on the file URL), searches, second pages and LinkQueryOnly case above. It failed with 27 problems on the previous files and passes now.

Results after the fix:

  • 6,200 runs; 6,185 send at least one request.
  • Recorded errors fell from 213 to 32. The ones left are real CLI responses, such as --from and --to are required, or bare-array lists given an object.
  • The most requests in any run is 4, under the cap of 20.

Checks:

  • Both golden tests pass alone, with -count=2, and in another timezone.
  • go vet ./..., go test -p 1 ./... and the 74 codegen tests pass.
  • The golden files hold no real identity or tenant data.

🤖 Generated with Claude Code

…ches

A command that shares its path with an earlier one is never run by Cobra, so
it is recorded as unreachable rather than with flag values a previous run left.
The fake server now sends a full first page and a short second one with matching
totals, a bare array to lists that expect one, Contact Center search pages, and
a default greeting purpose. Searches run with a time window and with --paginate.

The command tree adds --version itself and marks persistent flags, and the
download check reports an empty trace instead of panicking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@edumansky edumansky left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex review (gpt-6.2-astra, reasoning effort: high)

The golden tests pass, including repeated runs, but at least one committed trace records a request that a fresh CLI invocation cannot send. Several paging and search fixtures also stop before the requests they are meant to protect, and the command snapshot omits part of the published flag surface.

Comment thread cli_golden_test.go
full = append(full, "--token", "golden-token", "--organization", goldenOrg)
root := cmd.RootCommand()
root.SetArgs(full)
err := cmd.Execute()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Reset flags on the command Cobra actually dispatches

When two commands share a path, resetFlags(c) can reset a different command from the one cmd.Execute() selects. The committed get-priority-alert-2 run with no arguments sends a GET ending in /criteria/s-id, using the ID left over from the preceding run. A fresh CLI invocation instead fails with required flag(s) "id" not set. Resolve the dispatched command before resetting it, or isolate each invocation, so the snapshot represents what a user can actually send (AGENTS.md:37).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a. Five Calling commands share their path with an earlier command, for example call-settings-for-me get-priority-alert-2. Cobra always runs the first one. TestGoldenRequests now resolves each command with root.Find. When the command it finds is a different one, it records an unreachable run with no requests, so the snapshot no longer holds flag values left over from a previous run. TestGoldenFilesExerciseEachKindOfRequest checks that these runs record no requests.

Being unreachable is a real CLI bug. It is pinned here and will be raised separately.

Comment thread cli_golden_test.go
StatusCode: 200,
Header: header,
Body: io.NopCloser(strings.NewReader(fakeResponse)),
Request: req,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Return fixtures that exercise each list's paging contract

The same one-item JSON object cannot drive every pager. admin partner-tags get-all-customers-set expects a bare array and records item array is not an array before following its next link. calling numbers list-manage-jobs treats one item as a short final page, so its golden never records the next offset. Return a bare array where required and a full first page followed by a short page for pagers without totals. This tests the endpoint-specific behavior described in AGENTS.md:47 rather than pinning fixture errors or premature termination.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a. The recorder now counts each path's requests in a run:

  • Page one is a full page, of the requested max/pageSize/limit/count/size, or 100 by default.
  • Later pages hold one item.
  • Totals say one more item than page one, and two pages.
  • Lists that expect a JSON array get one: a run that fails with "is not an array" is run again in array mode.

Both examples now read a second page: admin partner-tags get-all-customers-set follows its link, and calling numbers list-manage-jobs asks for the next offset. 318 --paginate runs now record a later page. The regression test checks both commands by name.

Comment thread cli_golden_test.go
runs = append(runs, run(rec, c, "body", append(append([]string{}, required...), `--body={"golden":true}`)))
}
if hasGET(runs[1]) {
runs = append(runs, run(rec, c, "paginate", append(append([]string{}, required...), "--paginate")))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exercise default and paginated Contact Center search requests

None of the six structured cc search commands records its default query or its paginated requests. The required-only run lacks the time window and stops before sending anything; the all-flags run overrides --fields; and hasGET excludes GraphQL pagination because it uses POST. Supply fixed --from and --to values for the minimal search run and a two-page GraphQL response for its --paginate variant, so changes to default field selection or the next cursor/page appear in the request diff (AGENTS.md:37).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a.

  • Each structured cc search command's required-only run now has fixed --from/--to, so it records the default query.
  • A --paginate run is added for these commands.
  • The recorder answers POST …/search with a GraphQL page for every query type, with hasNextPage true on the first page only. So each search records its next-page request.

The regression test checks that all 6 searches read two pages.

Comment thread cli_golden_test.go
c.LocalFlags().VisitAll(func(f *pflag.Flag) {
gc.Flags = append(gc.Flags, goldenFlag{
Name: f.Name, Shorthand: f.Shorthand, Type: f.Value.Type(), Default: f.DefValue,
Usage: f.Usage, Required: isRequired(f), Hidden: f.Hidden,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Record whether command flags are inherited by subcommands

LocalFlags() includes both persistent flags declared on the command and non-persistent local flags, but the snapshot does not distinguish them. Changing the root --dry-run from persistent to local would remove it from API subcommands without changing commands.json; the request runs do not pass that flag either. Record persistence alongside the flag definition so the golden protects global flag availability (AGENTS.md:89).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a. Each flag in commands.json now records persistent when it is in the command's PersistentFlags(). TestGoldenCommandTreeHasPublishedRootFlags checks that the root --dry-run is recorded as persistent.

Comment thread cli_golden_test.go
walk = func(c *cobra.Command) {
// Cobra adds --help when a command first runs; adding it here keeps the
// tree the same whichever commands ran before.
c.InitDefaultHelpFlag()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Include the root version flag in the command snapshot

Executing the version subcommand does not initialize the root command's lazy --version flag. As a result, commands.json omits both webex --version and its -v shorthand, and removing that published flag would pass the golden test. Initialize Cobra's default version flag alongside its help flag before collecting local flags (AGENTS.md:37).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a. The tree walk now calls InitDefaultVersionFlag() next to InitDefaultHelpFlag(). So --version and its -v shorthand are recorded, and TestGoldenCommandTreeHasPublishedRootFlags checks both.

Comment thread cli_golden_test.go Outdated
}
for _, field := range listItemFields {
body[field] = []any{map[string]any{"id": "golden-1"}}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Let the default greeting lookup reach its upload request

The required-only cc agent-personal-greeting-files upload run stops after two lookup GETs because no fixture item has systemDefault: true. The all-flags run supplies the purpose ID explicitly, so neither run checks that the ID returned by the lookup reaches the multipart POST. Return a deterministic default purpose from the greeting-purpose route and verify the resulting upload, consistent with the request coverage in AGENTS.md:37.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a. Every fake list item now has systemDefault: true, so the lookup finds a default purpose. The required-only cc agent-personal-greeting-files upload run now sends its multipart POST with that purpose, and the regression test checks it.

Comment thread cli_golden_test.go
}
case r.Variant == "all-flags" && hasArg("--output="):
downloads++
last := r.Requests[len(r.Requests)-1]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Report an empty download trace instead of panicking

If a download fails before its first HTTP request, make golden writes an empty request slice and this check indexes it at -1. That aborts the validation run without reporting which command failed or its recorded error. Check the length before accessing the last request and report r.Command and r.Error, as the search check already does.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 19c729a. The download check now reports the command and its recorded error when the trace is empty, before it reads the last request.

@edumansky

Copy link
Copy Markdown
Contributor Author

Codex review, round 2: outcome

Codex (gpt-6.2-astra, reasoning effort: high) reviewed 8693db5 and made seven points. I checked each one against the code and the recorded runs, and confirmed it was real. All seven are fixed in 19c729a.

# Finding Outcome
1 [P2] Flags reset on a command Cobra doesn't dispatch Fixed. Commands Cobra can't reach are recorded as unreachable, with no requests. There are 5 of them.
2 [P2] One fixture can't drive every pager Fixed. Page one is full and later pages are short, totals match, and lists that expect an array get one. 318 list runs read page two, including both cited commands.
3 [P2] Search default and paginated requests not recorded Fixed. Searches run with a fixed time window, plus a --paginate run against a two-page GraphQL response.
4 [P2] Persistent flags not distinguished Fixed. commands.json records persistent.
5 [P3] Root --version missing Fixed. InitDefaultVersionFlag() is called, so --version/-v are recorded.
6 [P3] Default greeting lookup never reaches its upload Fixed. The fake items include systemDefault, so the required-only upload sends its file.
7 [P3] Empty download trace would panic Fixed. The check reports the command and its error.

Regression tests:

  • TestGoldenFilesExerciseEachKindOfRequest now also checks:
    • unreachable runs;
    • a second page for both cited lists;
    • two pages for every search;
    • the default-purpose greeting upload.
  • The new TestGoldenCommandTreeHasPublishedRootFlags checks --version -v and that --dry-run is persistent.
  • Both failed on the previous files and pass now.

Results after the fix:

  • 6,196 runs; 6,185 send at least one request. The other 11 are 5 unreachable commands and 6 runs that end in the CLI's real error for a missing or non-object body.
  • Recorded errors fell from 32 to 6.
  • The most requests in any run is 4.

Checks:

  • Both golden tests pass alone, with -count=2, and in another timezone.
  • go vet ./..., go test -p 1 ./... and the 74 codegen tests pass.
  • The golden files hold no real identity or tenant data.

Found while checking, not changed here:

  • The 5 commands Cobra can't reach are a CLI bug. They're pinned in the golden files and will be raised separately.
  • cc search --paginate returns the items it already has when a later page fails, instead of failing (internal/search/paginate.go). This is also pinned as is.

🤖 Generated with Claude Code

@edumansky
edumansky merged commit 0f1c0d0 into main Nov 17, 2026
1 check passed
@edumansky
edumansky deleted the golden-files branch November 17, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant