Repository navigation
Report per-case results in verify --json, and send a run environment on v2 reconcile #596
Description
This asks the CLI for two additive, optional pieces of data. The cloud server already accepts both and has tests for them. The parts of the dashboard and generator that read them are built, and they wait only on a CLI release that sends them. Tracked on the server side in taskless/taskless#334.
Neither field changes a verdict, an exit code or any existing output. A server that doesn't know a field ignores it, and a CLI that omits it gets the same behavior it gets today. The two can ship independently and in either order.
1. Per-case results in taskless verify --json
Ask: add an optional cases array to each rule in the verify --json report, with one entry per fixture case:
cases?: Array<{
name: string; // the case's path under tests/, e.g. "fail/cast"
expected: 'fail' | 'pass'; // should this case trigger the rule, or not?
ok: boolean; // did it behave as expected?
message?: string; // why it didn't, when ok is false
files?: Array<{ path: string; content: string }>; // the case's files, when cheap to include
}>Why: today a failed rule reports only errors: string[], and the case name appears only inside that prose (fail fixture did not fire: fail/<name>). The server deliberately does not parse it out, because that wording is the CLI's to change. So when the generator's sandbox runs verify on a rule it wrote and the rule fails, the dashboard's request timeline says "Failing case: not provided" every time. With cases, it can show which case failed, what was expected, and the files involved.
Expectation:
- The server reads
casesas structured data only, inworkers/generator/src/verify-harness/decide-from-report.ts, under design D13 of the rule-first dashboard. - Each entry with
ok: falseis stored as a failing case on the generation run, capped to a few cases per run. - A malformed entry is ignored rather than trusted.
- A report without
casesbehaves exactly as it does now. - The server code's comment expected this in 0.13.2. As of
@taskless/cli-nightly@0.13.5-20261030180659x0b41364the field is not emitted.
2. A run environment on POST /cli/api/v2/reconcile
Ask: include an optional environment object in the reconcile request body:
environment?: {
ci?: boolean; // true when running in a CI job rather than on a developer machine
provider?: string; // the CI provider when known, e.g. "github-actions"
}Why: the dashboard wants to show which repositories have Taskless running in CI: a "CI set up" dot in the sidebar, org home and switcher, lit after a CI reconcile within the last 30 days. Only the CLI knows whether a reconcile is running in CI. It already detects that, but uses it only for telemetry (envOS / ci / ciProvider), never in the reconcile body. Until it sends it, the server cannot tell, and the dot cannot ship. A dot backed by no data would read as "CI is off" for every repository.
Expectation:
- The field is in the published schema at
/cli/api/v2/__schema. On the server, it is atworkers/dashboard/app/routes/cli/api/v2/reconcile.tsx. - When
environment.ci === true, the server records the repository'slast_ci_reconcile_at. It does nothing else with the field. - Verdicts, entitlements and the response shape are identical with or without the field.
- Sending
ci: false(or omittingenvironment) on a developer machine is correct and expected. - Once a stable release sends this, the cloud side ships the dot (taskless/taskless#334, item 3).
Activity
theCodeDrift commentedon Nov 2, 2026
Shipped in #597 (
environmenton v2 reconcile) and #598 (casesinverify --json, for sg, vale and runtime). Both were checked against the server built in taskless/taskless#333, and neither needs a change on the cloud side.