[NachoEchevarria]

What Does This Do

Exercise instruction-change and agent-setup validation with one shared Java smoke case, using platform PR #48, pinned to 2ba4fe55.

Motivation

Verify Java onboarding and support for its existing symlinked skills.

Additional Notes

Live setup and instruction-change smoke runs passed with symlink support. One case validates the workflow, not setup value.

Jobs are restricted to this branch; setup is advisory. Remove the temporary enablement and AGENTS.md trigger before merging.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

[pr-commenter]

LLM Validation (setup smoke)

Agent Setup Evaluation — java-setup-smoke

Mode: agent setup. Baseline = minimal; candidate = default.

Setup conclusion: no_clear_difference · coverage: full.

Limited evidence: one selected case and one initial run per variant (confirmation trials may be added). This sample checks the evaluation workflow; it is insufficient evidence of setup or individual skill value.

✅ PASS

• Overall quality improved by 4.2 points with no blocking-case regressions.

1 case(s) · quality 81.7 → 85.8 (+4.2) · win-rate 50% [50%–50%]

Details — analysis and per-case results

Setup context

Both variants use the same commit and fresh state per agent attempt. This compares the declared repository setup, not the PR's instruction changes.
PASS does not prove benefit; no clear difference does not establish equivalence.

Commit: dc8a7ef57f296c291e804fc7ac0603f3904a08a3. Coverage: full. Selected cases: 1. Initial runs/case: 1. Completeness: complete.

Pilot verification: fresh checkout/configuration, declared instruction entry points/imports, and runtime skill discovery. Instruction context is not independently attested by the CLI; model compliance is judged separately.

Analysis

No safety or blocking-case regressions across 1 case(s). Overall pairwise win-rate 50% [50%–50%], quality +4.2 — see the verdict above for whether that clears the noise band.

Results

• Pairwise win-rate: 50% [50%–50%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
• Overall quality: 81.7 → 85.8 (/100, +4.2)
• Bad signals introduced (advisory): 0
• Candidate criteria coverage (advisory): 4/4 (100%) — expected_criteria the candidate met; does not affect the gate
• Blocking-case regressions: 0

_{Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.}

Platform commit: 745818b5d65805d5852b07cbdf4d09c29b3da6c9.

Job · Report · Answers and judgments · All artifacts

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.89 s	[+0.0%; +1.6%] (maybe worse)
startup:insecure-bank:tracing:Agent	12.83 s	12.95 s	[-1.5%; -0.2%] (maybe better)
startup:petclinic:appsec:Agent	17.17 s	17.03 s	[+0.2%; +1.5%] (maybe worse)
startup:petclinic:iast:Agent	16.87 s	17.11 s	[-2.3%; -0.6%] (maybe better)
startup:petclinic:profiling:Agent	16.17 s	16.95 s	[-8.7%; -0.4%] (maybe better)
startup:petclinic:sca:Agent	17.13 s	17.03 s	[-0.3%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.25 s	16.26 s	[-1.1%; +0.9%] (no difference)

Commit: dc8a7ef5 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[pr-commenter]

LLM Validation

LLM Validation Gate — java-setup-smoke

✅ PASS

• No blocking-case regressions; the quality change is within noise (baseline/candidate confidence intervals overlap).

1 case(s) · quality 87.5 → 85.0 (-2.5) · win-rate 50% [50%–50%]

Details — analysis and per-case results

Analysis

Changed instruction file(s): AGENTS.md.

No safety or blocking-case regressions across 1 case(s). Overall pairwise win-rate 50% [50%–50%], quality -2.5 — see the verdict above for whether that clears the noise band.

Results

• Pairwise win-rate: 50% [50%–50%] — candidate's share of blind comparisons (90% CI; spanning 50% = no clear difference)
• Overall quality: 87.5 → 85.0 (/100, -2.5)
• Bad signals introduced (advisory): 0
• Candidate criteria coverage (advisory): 4/4 (100%) — expected_criteria the candidate met; does not affect the gate
• Blocking-case regressions: 0

_{Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs.}