Skip to content

ci: propose the catalog refresh even when the limits generator fails - #168

Merged
viztor merged 4 commits into
mainfrom
claude/vigilant-mendel-bctz9n-refresh-decouple
Nov 12, 2026
Merged

viztor merged 4 commits into
mainfrom
claude/vigilant-mendel-bctz9n-refresh-decouple

Conversation

@viztor

@viztor viztor commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Why

The nightly catalog freshness job regenerated both vendor tables in one fail-fast step: the catalog first, then the Go limits. Since 2026-11-07 the limits generator has failed every night, because the vendor's Go page changed under its reader. So no refresh pull request has been proposed since 11-06, not even for the catalog. According to the 11-11 run's own log, the catalog moved Go 31 → 32 and Zen 82 → 87 in that time.

What changed

.github/workflows/ci.yml, the nightly "propose the refresh" step

  • Both generators are guarded the same way. A failure:
    • is annotated with ::error:: where it is detected, so a later failing command cannot swallow the cause;
    • leaves its file as committed (git checkout);
    • does not stop the other file's refresh, which still goes out;
    • fails the step at the end.
  • The catalog file is put back before the limits generator starts. That generator imports src/catalog-data.ts, so it never reads a half-written catalog.
  • When nothing changed, the log calls current only the files that actually regenerated.
  • A partial refresh replaces an open refresh PR's other file, with a warning. The branch is rebuilt from main every night. When one generator failed, the run warns that the open refresh PR no longer proposes the kept file. This is deliberate: an older night's output is never carried into a commit that says it was regenerated.
  • A failed gh pr create is no longer read as "already open". The step asks GitHub whether a refresh PR is open (gh pr list … --jq length = 1). Otherwise, for example on an expired token or an API error, the run fails rather than leaving a pushed branch that no PR proposes behind a green run.

test/freshness-scope.test.ts

A new describe finds the step by name in the parsed ci.yml and executes its run text under bash -e, in a scratch git repository with stubbed pnpm, vp fmt, git push and gh. It covers:

  • each generator failing;
  • both failing, and neither failing;
  • a failing push after a limits failure;
  • a failed gh pr create with and without an open PR.

The scratch repositories drop every inherited GIT_* variable, so an exported GIT_DIR cannot redirect the step's real git commands.

CONTRIBUTING.md

One bullet in "The freshness checks in CI" that states all of the above.

The reader fix is #167, which carries these three files byte for byte. It is separate because its own limits:shim gate needs a table regenerated from the live page.

Review

Two adversarial rounds. Each finding went to two skeptics, and only upheld findings were fixed.

  • Upheld and fixed:
    • the guard covered only one direction;
    • an annotation could be lost on a later failure;
    • nothing executed the step;
    • the partial-refresh replacement was unstated;
    • a misleading "both current" line;
    • the scratch repo inherited GIT_*;
    • gh failures passed for "already open".
  • Declined by both skeptics: making the commit title depend on which generator failed.

Verification

Node 26 on the pushed head 38e2d7f:

  • vp check: 0 errors and 21 warnings, the same as main.
  • Unit suite: 92 files, 1444 tests passed.
  • Mock suite: 52 passed.
  • pnpm run build and scripts/check.ts: check passed.
  • The step's run text, extracted from the YAML, passes bash -n.
  • Red first: against the previous step text, the new step tests failed on the catalog-failure, both-fail and failing-push cases, then passed after the fix.
  • No local paths in the diff.
  • Freshness scope for this diff: catalog:shim, limits:shim and cli:version all report.

🤖 Generated with Claude Code

https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

The nightly refresh step ran both generators in one fail-fast step, the
limits generator second. Since 2026-11-07 it has failed every night (the
vendor's Go page changed under its reader), so no refresh was proposed,
not even for the catalog, while the vendor added six models.

A limits failure now leaves src/go-limits-data.ts as it was. The catalog
refresh still goes out, and the step fails at the end, naming the cause,
so the signal is not lost. CONTRIBUTING says so beside the other
freshness rules.

The reader fix itself is separate, because its own freshness gate needs
a table regenerated from the live page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…y running it

ci-shell-2 / rules-docs-1: the catalog generator was still bare under
bash -e, so a catalog failure aborted the step before the limits
generator ran, while the step comment and CONTRIBUTING claimed the two
sources break independently. The catalog generator is now guarded the
same way as the limits one: a failure restores src/catalog-data.ts
before the limits generator starts (it imports that file, so it reads
the committed catalog), the limits refresh still goes out, and the step
fails at the end if either failed. The comment and the CONTRIBUTING
bullet now describe both directions.

ci-shell-4: each ::error:: is emitted inside its own failure block,
where the failure is detected, so a later git commit/push that fails
the step under bash -e cannot swallow the cause. Only the exit stays at
the end.

ci-shell-3: test/freshness-scope.test.ts finds the step by its name in
the parsed ci.yml and executes its run text under bash -e in a scratch
git repository, with pnpm, git push and gh stubbed: a limits failure
(catalog refresh committed, limits restored, limits named), a catalog
failure (limits generator reads the committed catalog, limits refresh
committed), both current (exit 0, no commit, no push), both failing
(exit 1, no commit, both named), and a failing push after a limits
failure (the cause is still annotated). Against the previous step text
the catalog-failure, both-failing and failing-push cases failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
ci1-1: the nightly refresh branch is rebuilt from main and force-pushed,
so a run in which one generator failed replaces an open refresh pull
request with a partial one, and an earlier night's change to the failed
file is no longer proposed. That is kept deliberately (an older night's
output is never carried into a commit that says it was regenerated), and
is now stated: the step records which files it kept as committed and,
once the push succeeds, emits a ::warning:: naming the branch and those
files. CONTRIBUTING.md's bullet says the same. test/freshness-scope.test.ts
gains scratch-repository cases with a pre-existing refresh branch: a
partial run replaces it (only the catalog differs from main) and warns
naming only the failed file; a complete run replaces it with no warning;
a failed push warns nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…re current

ci2-1: the nightly-step tests built every child environment from
process.env, so an inherited GIT_DIR, GIT_WORK_TREE or GIT_INDEX_FILE (a git
hook, or `git rebase --exec` in a linked worktree) sent the real git commands
to the repository they named: its branch, index and config were rewritten.
`gitIn` and `execute` now drop every GIT_* variable before adding their own,
and a case exports GIT_DIR, GIT_WORK_TREE and GIT_INDEX_FILE at a second
scratch repository, runs the step, and asserts that repository unchanged.

ci2-2: with one generator failed and nothing changed, the step logged "both
generated files are current" for a file it never regenerated. The message now
branches on the kept files ("not regenerated: <files>"); cases cover
fails/unchanged, unchanged/fails, and both failing.

ci2-3: `gh pr create ... || echo "already open"` read every gh failure as an
open pull request, so an expired token or an API error left a pushed branch
no pull request proposed behind a green run. The step now asks `gh pr list
--head chore/generated-refresh --state open`, and otherwise annotates the
failure and fails at the end. The gh stub takes the creation's and the
listing's outcomes; cases cover open, none open, an unanswerable listing,
a creation failure after a generator failure, and a created pull request.

CONTRIBUTING.md's nightly bullet says both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
viztor pushed a commit that referenced this pull request Nov 12, 2026
Make this branch carry exactly #168's versions of the files the two pull
requests share, so the nightly "propose the refresh" step and its tests
are one change wherever they land.

- .github/workflows/ci.yml: the nightly step now guards both generators
  the same way (each failure annotated where it happens, its file put
  back as committed, the other refresh still proposed, the step failing
  at the end), with the catalog restored before the limits generator
  starts. Byte-identical to #168.
- CONTRIBUTING.md: the "one broken source" bullet takes #168's text.
  This branch had no other CONTRIBUTING.md edit, so nothing was merged
  by hand. Byte-identical to #168.
- test/freshness-scope.test.ts: #168's tests that run the step itself in
  a scratch repository. Byte-identical to #168.

No findings carried in this commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
viztor pushed a commit that referenced this pull request Nov 12, 2026
…ghtly step's fixes

reader2-2: tableRows stripped tags and never decoded character references,
so `&#36;15` was no dollar amount (a valid page refused), `$15&nbsp;` read
as NaN, `Grok&nbsp;4.7` keyed as `grok-nbsp-4.7` and `Q&amp;A 2` as
`q-amp-a-2`, and names shipped with a literal `&gt;`. tableRows now decodes
after stripping tags and before collapsing whitespace, in one pass
(`&amp;lt;` stays the text `&lt;`): `&amp; &lt; &gt; &quot; &apos; &nbsp;
&le; &ge;` and numeric `&#N;`/`&#xN;`, leaving an unknown name (looked up
in a Map, so `&constructor;` is unknown) or a number that names no character
as written. The `&#36;15` case no longer stands for "no dollar amount";
new cases read entity-encoded amounts, key a label with `&nbsp;`, `&gt;`,
`&amp;` and `&le;`, decode in one pass and leave unknown references alone.
docs/testing.md's row says so.

reader2-3: firstDoc's JSDoc said a later source is asked only when every
earlier one carried neither shape; it is asked whenever the earlier ones
answered nothing the parser reads (a status outside 2xx, neither shape, or a
shape the parser rejected). Reworded.

ci2-1, ci2-2, ci2-3: .github/workflows/ci.yml, CONTRIBUTING.md and
test/freshness-scope.test.ts copied byte-identical from
#168 (no inherited GIT_* reaches the scratch-repo
tests; the "both current" line only when both regenerated; an open refresh
pull request is asked for, not assumed, and a failed creation fails the step).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
@viztor
viztor merged commit 17fe664 into main Nov 12, 2026
7 checks passed
viztor pushed a commit that referenced this pull request Nov 12, 2026
viztor pushed a commit that referenced this pull request Nov 12, 2026
… into claude/vigilant-mendel-bctz9n-p-h

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
viztor pushed a commit that referenced this pull request Nov 12, 2026
The stack's parent moved: #110 (P-T1b, c4fca32) and #112 (P-H, 1b4ebf0)
are on main, so P-R1 is rebased directly onto main 1.9.2 as its own
change plus its adaptation commits.

- faithful-2-F1 / hosts-2-R2-H1: fixed in the rewrite. The rebase
  commit's body no longer names a base release that can go stale; it
  reads "Rebased onto main as its own change, on top of P-H (#112)."
- rules-docs-2-D1: fixed. The branch drops P-H's pre-rebase history
  merge and everything below it, and sits on origin/main 1b4ebf0
  (conflict-free; the only tree change from the old base is #168). The
  plan's status table marks P-T1b "merged (c4fca32)" and P-H "merged
  (1b4ebf0)", phase 5 says P-T1b is merged, and P-R1's row and the
  "claim on both host cores" bullet say it is rebased onto main 1.9.2
  with P-H merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
viztor pushed a commit that referenced this pull request Nov 12, 2026
…9n-go-limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants