Repository navigation
Record the CLI's commands and requests as golden files - #30
Conversation
Every command's name, aliases, help and flags, and the exact request each API command sends for a fixed set of flags (all flags, required flags only, --body, --paginate), answered by a recording transport with no network. A change that is meant to keep the CLI the same must leave these files unchanged. make golden rewrites them, and the weekly collection refresh runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edumansky
left a comment
There was a problem hiding this comment.
Codex review (gpt-6.2-astra, reasoning effort: high)
The golden suite passes in its default order, but isolated and repeated runs fail on an unchanged checkout. Several request snapshots also pin fixture errors instead of the HTTP behavior they are intended to protect.
| var walk func(c *cobra.Command) | ||
| walk = func(c *cobra.Command) { | ||
| gc := goldenCommand{Path: c.CommandPath(), Aliases: c.Aliases, Short: c.Short, Long: c.Long, Hidden: c.Hidden} | ||
| c.LocalFlags().VisitAll(func(f *pflag.Flag) { |
There was a problem hiding this comment.
[P2] Make the golden tests independent of Cobra execution order
go test -run '^TestGoldenCommandTree$' . fails on an unchanged checkout: Cobra adds --help lazily, so the committed tree includes flags installed by TestGoldenRequests. Conversely, go test -run Golden -count=2 . fails because the second request sweep passes --help=true and prints help instead of sending requests. Initialize the command-tree flags deterministically and exclude help from request arguments so the golden check works independently and on repeated runs.
There was a problem hiding this comment.
Fixed in 8693db5. The command tree now calls InitDefaultHelpFlag() on each command before recording it, so it no longer depends on which commands ran first. The request sweep adds help to skippedFlags, so a second sweep no longer passes --help=true. These now pass:
go test -run '^TestGoldenCommandTree$' .go test -run '^TestGoldenRequests$' .go test -run Golden -count=2 .
| case "duration": | ||
| return "90s" | ||
| default: | ||
| return "s-" + f.Name |
There was a problem hiding this comment.
[P2] Record multipart uploads instead of missing-file errors
All 17 Calling and CC upload commands receive --file=s-file, which does not exist in the temporary directory. Every variant therefore records zero requests; changing an upload's method, route, or multipart field names would leave the goldens green. The recorded OS error also differs on Windows. Create a small fixture and record its multipart fields with the random boundary normalized so these commands meet the request snapshot contract.
There was a problem hiding this comment.
Fixed in 8693db5. Each run's working directory now holds golden.wav, which is the sample --file. The recorder replaces the multipart boundary with golden-boundary in the header and the body. All 17 upload commands now record their multipart request. The new TestGoldenFilesExerciseEachKindOfRequest fails if an upload sends none; it failed on the previous golden files.
| required = append(required, arg) | ||
| } | ||
| }) | ||
| runs := []goldenRun{run(rec, c, "all-flags", all), run(rec, c, "required", required)} |
There was a problem hiding this comment.
[P2] Exercise the flag-built Contact Center search requests
For the six convenience searches, the all-flags run stops at invalid JSON in --filter=s-filter, and the required-only run stops because neither --from nor --to is marked required. The only recorded POST uses --body, which bypasses query construction entirely. Supply a valid fixed time window, filter, and aggregation for these commands so a change to the GraphQL request built from their published flags is visible in the request snapshots.
There was a problem hiding this comment.
Fixed in 8693db5. The sample values are now --filter={"golden":true}, --aggregate=s-field:count:s-name, --interval=DAILY and --time-comparator=createdTime. With them, the all-flags run of each of the six searches records its flag-built GraphQL POST, and the regression test checks this. The required-only run still records --from and --to are required, because that is the command's real behaviour without those flags.
| // AI Agent Studio commands look up the agent's runtime token first. | ||
| if strings.HasSuffix(req.URL.Path, "/agents/") { | ||
| body = `{"objects":[{"id":"s-agent-id","bot_access_token":"golden-bot-token"}],"meta":{"total_count":1}}` | ||
| } |
There was a problem hiding this comment.
[P2] Record the second request of binary downloads
The {} metadata response makes the CC audio/greeting and Calling/Meetings recording downloads return before fetching the binary. Their snapshots would still pass if the download GET sent the Webex bearer token to a pre-signed URL, contrary to the binary download invariant. Return a synthetic download URL in the expected metadata field and record the follow-up GET, including its headers.
There was a problem hiding this comment.
Fixed in 8693db5. The fake response now carries url and temporaryDirectDownloadLinks, all on https://download.golden.example/. Each of the 4 download commands now records the follow-up GET with its headers. The regression test fails if that GET is missing, or if it carries an Authorization header.
| return &http.Response{ | ||
| StatusCode: 200, | ||
| Header: http.Header{"Content-Type": {"application/json"}}, | ||
| Body: io.NopCloser(strings.NewReader(body)), |
There was a problem hiding this comment.
[P2] Exercise continuation requests in pagination snapshots
The recorder never supplies a second page, so the paginate snapshots cannot detect changes that affect only continuation requests. For example, removing LinkQueryOnly from admin resource-memberships list-group-v2 would change the URL used for page two without changing its golden. Use bounded two-page fixtures for the supported paging behaviors, including a next link whose path differs for this override.
There was a problem hiding this comment.
Fixed in 8693db5. The fake response now serves two pages:
- one item under each item field the lists read;
- totals of two items or two pages;
- on the first request only, a
Linkto/golden-next?golden_page=2, a different path.
296 --paginate runs now record a second page. The regression test also checks that admin resource-memberships list-group-v2 keeps its own path on page two, which is its LinkQueryOnly behaviour. Every run is capped at 20 requests, so a list can't loop forever; the most any run makes is 4.
| t.Helper() | ||
| home := t.TempDir() | ||
| t.Setenv("HOME", home) | ||
| t.Setenv("XDG_CONFIG_HOME", filepath.Join(home, ".config")) |
There was a problem hiding this comment.
[P2] Isolate the Windows user profile during golden runs
On Windows, appconfig.Dir() uses os.UserHomeDir(), which reads USERPROFILE, not HOME. These tests consequently load the developer's real Webex config, so a saved region can change the recorded hosts; aiagent.PrivateAPINotice() can also write its marker into the real profile. Set USERPROFILE to the temporary home as the existing cmd/ccregion_test.go isolation helper does.
There was a problem hiding this comment.
Fixed in 8693db5. goldenEnv now sets USERPROFILE to the temporary home as well as HOME, as cmd/ccregion_test.go does.
| var out []goldenCommand | ||
| var walk func(c *cobra.Command) | ||
| walk = func(c *cobra.Command) { | ||
| gc := goldenCommand{Path: c.CommandPath(), Aliases: c.Aliases, Short: c.Short, Long: c.Long, Hidden: c.Hidden} |
There was a problem hiding this comment.
[P2] Pin positional argument syntax in the command tree
CommandPath() includes only the command names, not the argument syntax in Cobra's Use field. For example, changing config set <key> <value> to config set <value> <key> would change the published usage without changing either golden test. Include Use or the usage line in the snapshot to cover the published help contract.
There was a problem hiding this comment.
Fixed in 8693db5. Each command in commands.json now records use, cobra's Use line with its argument syntax.
| if err != nil { | ||
| t.Fatalf("%v (run make golden to create it)", err) | ||
| } | ||
| if !bytes.Equal(want, buf.Bytes()) { |
There was a problem hiding this comment.
[P3] Keep golden comparisons stable on CRLF checkouts
With core.autocrlf=true, Git checks out these JSON files with CRLF, while json.Encoder emits LF on every platform. The byte comparison then rejects every snapshot even when the CLI is unchanged. Pin testdata/golden/*.json to LF in .gitattributes or normalize line endings before comparison.
There was a problem hiding this comment.
Fixed in 8693db5. The new .gitattributes pins testdata/golden/*.json to eol=lf, and git check-attr confirms it applies.
The recorded runs stopped at fixture errors: a missing upload file, a search filter that is not JSON, download metadata with no link, and lists with one page. A shared fake response now carries a second page, download links and the agent list, and the sample values satisfy the search and upload commands. Multipart boundaries are fixed so uploads record stably. The command tree records each command's usage and adds --help itself, and the request runs leave --help out, so the tests pass alone and repeated. Windows' USERPROFILE is isolated like HOME, and the golden JSON stays LF on checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex review: outcomeCodex (
Regression test: Results after the fix:
Checks:
🤖 Generated with Claude Code |
…ches A command that shares its path with an earlier one is never run by Cobra, so it is recorded as unreachable rather than with flag values a previous run left. The fake server now sends a full first page and a short second one with matching totals, a bare array to lists that expect one, Contact Center search pages, and a default greeting purpose. Searches run with a time window and with --paginate. The command tree adds --version itself and marks persistent flags, and the download check reports an empty trace instead of panicking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
edumansky
left a comment
There was a problem hiding this comment.
Codex review (gpt-6.2-astra, reasoning effort: high)
The golden tests pass, including repeated runs, but at least one committed trace records a request that a fresh CLI invocation cannot send. Several paging and search fixtures also stop before the requests they are meant to protect, and the command snapshot omits part of the published flag surface.
| full = append(full, "--token", "golden-token", "--organization", goldenOrg) | ||
| root := cmd.RootCommand() | ||
| root.SetArgs(full) | ||
| err := cmd.Execute() |
There was a problem hiding this comment.
[P2] Reset flags on the command Cobra actually dispatches
When two commands share a path, resetFlags(c) can reset a different command from the one cmd.Execute() selects. The committed get-priority-alert-2 run with no arguments sends a GET ending in /criteria/s-id, using the ID left over from the preceding run. A fresh CLI invocation instead fails with required flag(s) "id" not set. Resolve the dispatched command before resetting it, or isolate each invocation, so the snapshot represents what a user can actually send (AGENTS.md:37).
There was a problem hiding this comment.
Fixed in 19c729a. Five Calling commands share their path with an earlier command, for example call-settings-for-me get-priority-alert-2. Cobra always runs the first one. TestGoldenRequests now resolves each command with root.Find. When the command it finds is a different one, it records an unreachable run with no requests, so the snapshot no longer holds flag values left over from a previous run. TestGoldenFilesExerciseEachKindOfRequest checks that these runs record no requests.
Being unreachable is a real CLI bug. It is pinned here and will be raised separately.
| StatusCode: 200, | ||
| Header: header, | ||
| Body: io.NopCloser(strings.NewReader(fakeResponse)), | ||
| Request: req, |
There was a problem hiding this comment.
[P2] Return fixtures that exercise each list's paging contract
The same one-item JSON object cannot drive every pager. admin partner-tags get-all-customers-set expects a bare array and records item array is not an array before following its next link. calling numbers list-manage-jobs treats one item as a short final page, so its golden never records the next offset. Return a bare array where required and a full first page followed by a short page for pagers without totals. This tests the endpoint-specific behavior described in AGENTS.md:47 rather than pinning fixture errors or premature termination.
There was a problem hiding this comment.
Fixed in 19c729a. The recorder now counts each path's requests in a run:
- Page one is a full page, of the requested
max/pageSize/limit/count/size, or 100 by default. - Later pages hold one item.
- Totals say one more item than page one, and two pages.
- Lists that expect a JSON array get one: a run that fails with "is not an array" is run again in array mode.
Both examples now read a second page: admin partner-tags get-all-customers-set follows its link, and calling numbers list-manage-jobs asks for the next offset. 318 --paginate runs now record a later page. The regression test checks both commands by name.
| runs = append(runs, run(rec, c, "body", append(append([]string{}, required...), `--body={"golden":true}`))) | ||
| } | ||
| if hasGET(runs[1]) { | ||
| runs = append(runs, run(rec, c, "paginate", append(append([]string{}, required...), "--paginate"))) |
There was a problem hiding this comment.
[P2] Exercise default and paginated Contact Center search requests
None of the six structured cc search commands records its default query or its paginated requests. The required-only run lacks the time window and stops before sending anything; the all-flags run overrides --fields; and hasGET excludes GraphQL pagination because it uses POST. Supply fixed --from and --to values for the minimal search run and a two-page GraphQL response for its --paginate variant, so changes to default field selection or the next cursor/page appear in the request diff (AGENTS.md:37).
There was a problem hiding this comment.
Fixed in 19c729a.
- Each structured
cc searchcommand's required-only run now has fixed--from/--to, so it records the default query. - A
--paginaterun is added for these commands. - The recorder answers
POST …/searchwith a GraphQL page for every query type, withhasNextPagetrue on the first page only. So each search records its next-page request.
The regression test checks that all 6 searches read two pages.
| c.LocalFlags().VisitAll(func(f *pflag.Flag) { | ||
| gc.Flags = append(gc.Flags, goldenFlag{ | ||
| Name: f.Name, Shorthand: f.Shorthand, Type: f.Value.Type(), Default: f.DefValue, | ||
| Usage: f.Usage, Required: isRequired(f), Hidden: f.Hidden, |
There was a problem hiding this comment.
[P2] Record whether command flags are inherited by subcommands
LocalFlags() includes both persistent flags declared on the command and non-persistent local flags, but the snapshot does not distinguish them. Changing the root --dry-run from persistent to local would remove it from API subcommands without changing commands.json; the request runs do not pass that flag either. Record persistence alongside the flag definition so the golden protects global flag availability (AGENTS.md:89).
There was a problem hiding this comment.
Fixed in 19c729a. Each flag in commands.json now records persistent when it is in the command's PersistentFlags(). TestGoldenCommandTreeHasPublishedRootFlags checks that the root --dry-run is recorded as persistent.
| walk = func(c *cobra.Command) { | ||
| // Cobra adds --help when a command first runs; adding it here keeps the | ||
| // tree the same whichever commands ran before. | ||
| c.InitDefaultHelpFlag() |
There was a problem hiding this comment.
[P3] Include the root version flag in the command snapshot
Executing the version subcommand does not initialize the root command's lazy --version flag. As a result, commands.json omits both webex --version and its -v shorthand, and removing that published flag would pass the golden test. Initialize Cobra's default version flag alongside its help flag before collecting local flags (AGENTS.md:37).
There was a problem hiding this comment.
Fixed in 19c729a. The tree walk now calls InitDefaultVersionFlag() next to InitDefaultHelpFlag(). So --version and its -v shorthand are recorded, and TestGoldenCommandTreeHasPublishedRootFlags checks both.
| } | ||
| for _, field := range listItemFields { | ||
| body[field] = []any{map[string]any{"id": "golden-1"}} | ||
| } |
There was a problem hiding this comment.
[P3] Let the default greeting lookup reach its upload request
The required-only cc agent-personal-greeting-files upload run stops after two lookup GETs because no fixture item has systemDefault: true. The all-flags run supplies the purpose ID explicitly, so neither run checks that the ID returned by the lookup reaches the multipart POST. Return a deterministic default purpose from the greeting-purpose route and verify the resulting upload, consistent with the request coverage in AGENTS.md:37.
There was a problem hiding this comment.
Fixed in 19c729a. Every fake list item now has systemDefault: true, so the lookup finds a default purpose. The required-only cc agent-personal-greeting-files upload run now sends its multipart POST with that purpose, and the regression test checks it.
| } | ||
| case r.Variant == "all-flags" && hasArg("--output="): | ||
| downloads++ | ||
| last := r.Requests[len(r.Requests)-1] |
There was a problem hiding this comment.
[P3] Report an empty download trace instead of panicking
If a download fails before its first HTTP request, make golden writes an empty request slice and this check indexes it at -1. That aborts the validation run without reporting which command failed or its recorded error. Check the length before accessing the last request and report r.Command and r.Error, as the search check already does.
There was a problem hiding this comment.
Fixed in 19c729a. The download check now reports the command and its recorded error when the trace is empty, before it reads the last request.
Codex review, round 2: outcomeCodex (
Regression tests:
Results after the fix:
Checks:
Found while checking, not changed here:
🤖 Generated with Claude Code |
The CLI's published surface is now recorded in golden files, and a test fails if any of it changes.
This is PR 1 of the SDK extraction plan. PR 2 will move the CLI's API logic into
pkg/. These files will show that the move leaves the CLI the same.What's recorded
testdata/golden/:commands.json: every command's path, aliases, help and flags, with each flag's type, default, usage, and whether it's required or hidden.requests-<area>.json, for each of the six API areas: the exact requests each command sends, made through a recording transport with no network. Each request has its method, URL, headers and body.Each command is run with:
--body, if it takes one;--paginate, if it sends a GET.The fake server answers like a two-page list:
It also has download links, the agent list AI Agent Studio commands look up, a default greeting purpose, and Contact Center search pages.
The sample values satisfy the upload and search commands, and searches run with a time window and with
--paginate. Multipart boundaries are fixed. So uploads, downloads with their file GET, searches and later pages are all recorded.A command that shares its path with another one is never run by Cobra, so it is recorded as
unreachable. A run that still fails is recorded with its error, which is the CLI's real response to those flags.commands.jsonalso records each command'suseline,--version, and which flags are persistent.Changes
cli_golden_test.go:TestGoldenCommandTreeandTestGoldenRequests.HOMEandUSERPROFILE, the file token store so the OS keychain is never read, clearedWEBEX_*variables, and a fake token and organization.TestGoldenFilesExerciseEachKindOfRequestchecks that the files really contain the uploads, downloads (with no token on the file's URL), searches with their pages, later list pages and unreachable commands.TestGoldenCommandTreeHasPublishedRootFlagschecks--versionand the persistent root flags..gitattributeskeeps the golden JSON LF on every checkout.make goldenrewrites the files after an intended change.make golden, so its PR shows each command that changed.AGENTS.mddescribes the files and when to update them.This is #20's
36ef6fa, adapted tomain. It hasmain's environment variables, the file token store, and the AI Agent Studio agent list in its paged form.Checks
--paginateruns read a later page.-count=2, and in other timezones, andmake goldenchanges nothing.IdentityBaseURLtohttps://webexapis.com/v1made the test fail on thearchive-usersURL, naming the line.golden-token.go vet ./...,go test -p 1 ./...and the 74 codegen tests pass.🤖 Generated with Claude Code