Skip to content
NickAme03Public

About

Una prova sul distinguere, in un diff, un difetto introdotto dal cambiamento da uno preesistente. La gold si deriva eseguendo, non si scrive a mano.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

eval-diff

Given a change to some code and a review finding written about that change: does the finding report a defect introduced by the change, or one that was already there?

Headline result. One case is wrong in 9 runs out of 9 when the reviewer sees only a diff fragment and the finding blames the wrong thing, and right in 12 runs out of 12 as soon as either the surrounding code or an accurate finding is given back. Same engine, same prompt, only the input condition changes. The gold labels are derived by executing the code, never written by hand.

Those are two different questions, and a review bot almost always answers the first one, because the diff is the only thing it can see.

The eval comes from a real case. On 2026-07-31 a review bot on one of my pull requests called a behaviour a regression. That behaviour was byte for byte identical in the target branch, so the change could not have introduced it. The correct answer was "already there".

The Italian version of this document, which goes into more detail, is README.it.md. The four result write-ups are in Italian as well.

The one property that matters

The gold labels are not written by hand. Every case has two versions of a small file, base and head, plus a check.py that exercises the behaviour the finding complains about. build_gold.py runs the check against both and derives the label:

base head label
ok fails introduced_by_diff
fails fails pre_existing
ok ok excluded, the check does not demonstrate the finding
fails ok excluded, the change repairs it

The two excluded rows are deliberate: cases the definition cannot decide are dropped from the dataset instead of receiving a third label.

So no label is anyone's opinion. If you disagree with a case, re-run it and the argument is over. That is the whole point, and it is why the expensive design was chosen over two cheaper ones. The reasoning, written before any code, is in definizione-etichette.md.

The positional floor

There is a shortcut: answer introduced_by_diff whenever the finding points at a line the diff touched. The cases are built so that this shortcut scores 0.40. A result that does not beat 0.40 did not read the code, it compared line numbers.

That number is a design choice, not a discovery, and it is stated so nobody mistakes it for a property of the task.

What it found

Ten cases, five small and five larger with several changes of which only one is relevant. Twenty-one runs on this dataset, plus thirteen earlier ones on the five-case version, with every prompt and raw answer saved on disk so anyone can check what the model actually saw. tabella_corse.py rescores all of them from the files.

The engine under test is Codex gpt-5.6-sol at two reasoning-effort levels.

condition runs accuracy
fragment view, misattributed finding, low effort 6 0.70 to 0.90
fragment view, misattributed finding, default effort 3 0.90
fragment view, accurate finding, low effort 3 1.00
whole files, misattributed finding, low effort 3 1.00
whole files, accurate finding, default effort 6 1.00

One case, case-06-modifica-lontana, is wrong in 9 runs out of 9 whenever the view is a fragment and the finding blames the wrong thing, at either effort level, and is right in 12 out of 12 everywhere else.

It is an interaction, not a single cause. Give the model back either the surrounding code or an accurate finding and it recovers. It needs to lose both.

Every error, in all sixty answers of the weak condition, goes the same way: pre_existing where the truth is introduced_by_diff. The implicit rule is readable from the data:

the function named by the finding did not change, therefore the problem predates the change.

That is the failure of the founding case with the sign flipped. Both are the same mistake: judging whether the cited code changed instead of whether the behaviour changed.

What it does not prove

Stated here rather than buried, because the numbers are small.

  • One engine. Every number above comes from gpt-5.6-sol. I attempted a cross-family comparison on 2026-08-01 across four setups, and it produced no usable table: all four failed the admissibility gate that had been written beforehand, getting the strong condition wrong too. None of those twelve runs enters a figure above. The account is in esito-famiglie.md, the gate in criterio-abbandono-famiglie.md. The limit stands; the machinery to close it is in place.

    Those figures measure setups, not models, and they do not compare with each other. Each setup runs at its own reasoning regime, declared in the sforzo column of tabella_corse.py, and two of them go through a different harness than codex exec. The data says "this setup does not clear the gate", never "this model cannot do the task". tabella_corse.py prints the same warning on every run.

    The reasoning imbalance was measured, and it does not carry the result. Codex with reasoning off scores 1.00 across four runs of the strong condition, forty answers and zero errors, exactly as codex reasoning does. The gate's bar therefore never required reasoning, so withholding it from the other setups did not withhold the bar.

    Two of those four are "undetermined", not failed. Three runs over ten cases leave an uncertainty band about 0.32 wide, and for two setups that band straddles the gate's threshold. py -3 fascia_tassi.py recomputes every band and prints a three-way verdict: below, above, or undetermined.

  • Ten cases. Stable across six repeats per condition, but narrow coverage.

  • The failing family is over-represented. All three cases that fail have the shape "the named function is untouched". What the eval measures reliably is that specific failure mode, not diff review in general.

  • Read this one before the others: the dataset is calibrated on gpt-5.6-sol. criterio-abbandono-scala.md says a case that fails even in the strong condition should be suspected as an ambiguous fixture and excluded, which makes that engine's strong condition the working definition of "a fair case". The large cases were built by watching where it broke: case 06 puts 35 lines between the change and the finding because 35 was the distance that broke it. So these ten cases cannot rank engines. A different engine is not measured in absolute terms, it is measured on how closely it resembles the one the set was tuned against. That exclusion rule never fired and no fixture was ever dropped, but the calibration is in the design. The headline result, case 06, compares two conditions of the same engine and is untouched by it.

  • It measures an input condition, not a product. The defensible claim is that a reviewer who sees only a fragment and receives a misattributed finding fails in a specific way. Not that any particular tool is bad.

The abandonment criterion for the larger dataset was written before any of the larger fixtures existed, in criterio-abbandono-scala.md. It asked for a case wrong in at least 3 runs out of 6 in the weak condition and at most 1 out of 6 in the strong one. Both clauses were met. It is there so the thresholds could not be adjusted after seeing the numbers.

Running it

py -3 verifica_casi.py      # case invariants, recomputes the positional flag from the real diff
py -3 build_gold.py         # derives gold.json by execution
py -3 test_scorer.py        # tests the scorer
py -3 scorer.py esempi/risposte-perfette.json
py -3 tabella_corse.py      # every run, rescored from the files

Answers are a flat JSON object, {"case_id": "label"}. No rationales: the eval measures the classification and nothing else.

run_model.py presents the cases to a model. Its levers move one at a time, because otherwise a change in score cannot be attributed to anything: --vista intera|frammento, --rilievo preciso|impreciso, and --config k=v for reasoning effort.

--motore codex|oss|claude switches engine: oss routes the same prompts to a local model served by ollama through the same codex exec harness, claude routes them to the Claude Code command line.

The prompt is byte-identical across all three, verified with diff -r. What is not identical is everything around it: reasoning effort differs per engine and claude brings its own system prompt. So the switch does not produce comparable columns, and the runs table says so on every execution. Compare within one engine, across its input conditions. That is what this eval measures.

MIT licensed.

About

Una prova sul distinguere, in un diff, un difetto introdotto dal cambiamento da uno preesistente. La gold si deriva eseguendo, non si scrive a mano.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages