Repository navigation
Add triage-ci-failures skill for red PR checks - #15336
Merged
Merged
Conversation
A dependency bump like Pandoc 3.13 turns a PR red across dozens of bucket x OS jobs, each log several hundred KB. Check-run annotations only name the bucket, and `gh run view --log` silently truncates near 750 KB, so finding which tests failed meant pulling raw logs into context. The script takes a PR number (every failing run on the head commit) or a run id, caches failed-job logs per job id via the jobs/ID/logs endpoint, and prints one row per unique failing test with the OSes it failed on and the verify message. When verify hits a missing output file, the row shows the render error from the OUTPUT tail instead. `##[error]` lines are reported only for jobs with no failing test, since smoke jobs echo each failed bucket there. Jobs whose log shape is not parsed (Playwright, ff-matrix are unverified) are listed with their log path. The skill then has the agent group rows by cause and classify each as dependency output drift or a regression before touching assertions or snapshots. On PR #15320 it reports the 18 failing tests plus the runtime-boundary error in about 6 KB.
- Capture the assertion message up to the stack trace (max 6 lines). noErrors and noErrorsOrWarnings put the actual errors after a generic first line, which the digest dropped. - Treat "No output created message" and "Path ... doesn't exist" as a failed render, like an OS NotFound, and show the OUTPUT tail. - A FAILED test with no TEST FAILURE block (prereq or teardown-only failure, which test.ts rethrows undecorated) now names the log to grep instead of printing an empty row. - Download logs to a .part file and rename on success, so a failed fetch is retried next run instead of being cached truncated. The PR #15320 digest is unchanged. The new shapes were checked on a log written in the test.ts failure format.
A Sass failure reaches noErrorsOrWarnings as "Theme file compilation failed:\n\n<diagnostics>" (src/core/dart-sass.ts), and the diagnostics start with "Error:". Stopping the message at a blank line, or at any "Error:" line, dropped the compiler error. The message now ends at the stack trace, recognised as "at" frames or as "<ErrorName>: <first message line>", the line JS stacks begin with. Blank lines are skipped. XPath and layout verifiers append the whole document after "... follows:", so capture ends at that line; the blank line had been doing that job before. The PR #15320 digest only gains the "Path:" line of the NotFound message.
test-smokes.yml names jobs "Run smoke (<os>)" plus an optional " with timed file" or " - with some extra R packages". Requiring the parenthesis at the end of the name made those rows show the full job name instead of the OS.
A merged test row can contain failures with different causes. Identify the representative excerpt's OS and job and list all contributing logs so those differences can be checked before grouping failures by cause. Clarify that missing output and failures limited to one OS require investigation before assigning a cause. Verified on PR #15320: 18 unique failing tests and the runtime-boundary error, with the render diagnostics preserved in an 8.5 KB digest. Shell syntax and diff checks pass.
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A dependency bump such as Pandoc 3.13 (#15320) turns a PR red across dozens of test bucket and OS jobs, each log several hundred KB. Check-run annotations only name the bucket, and
gh run view --logsilently truncates near 750 KB, so finding which tests failed meant pulling raw logs into the agent's context.This adds a project skill,
triage-ci-failures, with a script that does the digging:jobs/ID/logsendpoint, cached per job id.The skill then has the agent group rows by cause and classify each cause as dependency output drift or a regression before changing any assertion or snapshot. Updating a snapshot to match new output is the easy move on a bump PR, so the skill makes the verdict explicit first.
On #15320 the digest reports the 18 unique failing tests and the runtime-boundary error in about 8.5 KB.
Checklist
I have (if applicable):
AI-assisted PR
Note: autonomous AI agents submitting PRs without human oversight are not permitted — see the Code of Conduct.