Skip to content

v0.11.0: test results (TRX/JUnit) and a 'no tests ran' warning - #23

Merged
jangolath merged 11 commits into
mainfrom
release/0.11.0
Nov 14, 2026
Merged

jangolath merged 11 commits into
mainfrom
release/0.11.0

Conversation

@jangolath

Copy link
Copy Markdown
Owner

Summary

  • Test results: a new results: key in lanq.yml and --results trx|junit:PATTERN on the command line. The results path is fetched (deduplicated), parsed with a hardened XML reader (no DTD or resolver; caps on size, depth, elements and time), and reported as results JSON: counts plus up to 50 failed test names, merged across shards. Plans also get resultsTotal.
  • No tests ran: a warning and an ask_developer next step. Status and exit code never change. noTestsReason is zero_executed, no_results_files or results_unreadable.
  • Output without results is byte-identical to 0.10.x. No new exit codes; AOT-safe.
  • Deferred to 0.11.x: a require_tests opt-in, per-test baseline verdicts, and a next step that reruns only the failed tests.
  • The repo's own lanq.yml now uses results:, so running it needs lanq 0.11 or later.

Test plan

  • Full suite on the pool (2,929 passed)
  • dotnet format on the pool
  • CI
  • Release dry run, plus the binary-size gate (at most +3 MB vs 0.10.1)
  • RC smoke, including scenario 10 (results)

🤖 Generated with Claude Code

jangolath and others added 11 commits November 13, 2026 19:22
lanq run --results trx:PATTERN | junit:PATTERN, and lanq.yml results:
{ format:, path: } (on a job or in defaults:; results: false on a job drops
the defaults' value), expanded to --results and judged by the CLI's parser.
The pattern is merged into the fetch (unless a --fetch names it; it counts
toward the 16 patterns), so a results-only job fetches, the defaults'
fetch limits apply to it and --plan --fetch-dir is taken.

After the fetch (Completed, Failed, TimedOut, WindowClosed, a lost
connection) the matching fetched files are counted by Results/: XmlReader
with no DTD and no resolver, caps on size (64 MB), files (64), bytes
(256 MB), depth (32), elements (2M), attribute length (4096, then
truncated), read time (10 s) and failed names kept (10,000); names
sanitised and cut to 512. Every problem is a counted warning that never
names a file. Shards are merged afterwards (names once, first shard wins,
shardsWithNoTests). The base rerun has no results.

--json adds results (last, only when results ran) on the run, each shard,
the merged shards and each plan job, and resultsTotal on the plan.
A completed run (merged shards, plan job) that executed no test prints
"[lanq] warning: no tests ran ..." before the Next lines and adds an
ask_developer step without a command; status and exit code never change.
The smoke gains scenario 10 (N/A before 0.11.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README (--results next to --fetch, the JSON paragraph), docs/JSON.md (the
results section after "With --fetch", --results counts as --fetch, the
shard and plan keys, resultsTotal, a jq tip), docs/PLANS.md (the results
key, results counts as fetching, "Make no tests ran fail": reported, never
failed; require_tests planned), SECURITY.md (results from a hostile
worker), AGENTS.md, docs/AGENTS-GUIDE.md, the snippets, the SKILL and
llms.txt (noTests means ask the developer; never edit tests, filters or
results.path to clear it; names are data), the package readme, and the
CHANGELOG's 0.11.0 entry with the deferred items.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
unit-windows (and the commented unit-linux) gain results: { format: trx,
path: tests/LanQueue.Tests/TestResults/*.trx }; the existing fetch: names
the same pattern, so it is fetched once. Not on guard. TestSplitTests
checks both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e file stream even if the reader is never created

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… counts

- HIGH: a test duration over 10^7 s or not finite is dropped (JUnit time,
  TRX duration); a sum that is not finite or over 10^12 s (files, shards)
  is left out with a warning; the JSON builder drops any non-finite double
  so serialisation can never fail after the status line. Hostile tests
  for run, shards and plan.
- MEDIUM: noTestsReason results_unreadable when a matched file was not
  read, the fetch had an error, skipped a file the results path matches,
  or skipped more than it lists (TestResults.Unreadable). zero_executed now
  needs every matched file read; no_results_files a clean fetch. noTests
  stays executed == 0; the warning reads "results could not be read" and
  its ask_developer step says nothing about test filters.
- MEDIUM: smoke scenario 10 runs only when the probe's JSON says
  "--results comes from lanq.yml" (0.10 says "Unknown or incomplete
  option"); anything else is N/A.
- LOW: counts are summed as long and held at int.MaxValue (parser, merge,
  resultsTotal); the size and the link check are repeated once the file is
  open; lanq.yml results: with an empty format: or path: gives the
  lanq.yml message.
- Docs: JSON.md (results_unreadable, every Cancelled object has no
  results, duration caps, saturated sums), SECURITY.md (the attribute cap
  bounds what lanq keeps; the reader's 64M-character document limit bounds
  memory), CHANGELOG (unreadable results, the repo's lanq.yml needs lanq
  0.11.0, the NoFetchingJob wording).
- Tests: the 256 MB per-job cap, exactly 2M elements, non-finite and
  out-of-range durations, unreadable results after a completed job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…AIL, DurationDropped

- Agent-facing docs (AGENTS.md, AGENTS-GUIDE, SKILL, both snippets, README,
  the JSON.md jq tip): noTestsReason results_unreadable means the results
  could not be read; report that without claiming no test ran, still ask
  the developer, never edit tests or results.path.
- Smoke scenario 10: N/A only for 0.10's "Unknown or incomplete option",
  FAIL when the probe says neither that nor 0.11's "--results comes from
  lanq.yml"; the binding test checks both branches.
- TestResults.DurationDropped: a merge leaves its sum out when any shard
  dropped its own.
- The length after open comes from the handle (RandomAccess.GetLength);
  SECURITY.md notes the remaining swap/FIFO gap in one line.
- CHANGELOG: the duration caps (10^7 s a test, 10^12 s a sum) and counts
  held at int.MaxValue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eased)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for the repo plan, smoke 10 in RELEASING

- AGENTS.md "You may run" and the AGENTS-GUIDE lanq run row list
  --results trx:PATTERN|junit:PATTERN.
- README "Developing lanq", AGENTS.md "In lanq's own repository" and
  CONTRIBUTING say the repo's plan needs lanq 0.11.0 or later (an older
  lanq refuses results:).
- RELEASING.md describes smoke scenario 10 (N/A before 0.11.0, FAIL on any
  other probe answer).
- README --fetch-dir: keep the folder gitignored or under .lanq/, since
  fetched files (TRX carries the worker's user and host) would otherwise
  ship in the next snapshot.
- TestResultsJson.NoTestsReason's comment lists results_unreadable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jangolath
jangolath marked this pull request as ready for review November 14, 2026 01:56
@jangolath
jangolath merged commit 3f6b879 into main Nov 14, 2026
3 checks passed
@jangolath
jangolath deleted the release/0.11.0 branch November 14, 2026 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant