Repository navigation
FIX: Re-measure objective scorers so the default objective scorer is F1-ranked - #4234
Open
Richard Lundeen (richlundeen) wants to merge 1 commit into
Open
Richard Lundeen (richlundeen) wants to merge 1 commit into
Richard Lundeen (richlundeen) wants to merge 1 commit into
Conversation
Re-measure inverted_refusal (5d9c2551, gpt-5.4) and fallback (97da9dca,
gpt-4o) with three trials on all 14 objective gold files, and add first
metrics for main (0bba4802, gpt-4o-unsafe). Each entry now records
num_input_responses == num_responses == 395, so ScorerInitializer can rank
them for the default objective scorer, which moves from acs_with_refusal
(F1 0.366) to inverted_refusal (F1 0.885).
Remove the placeholder twin 19c69859, recorded under the unresolved
{AZURE_OPENAI_GPT5_4_UNDERLYING_MODEL} name; the re-measured 5d9c2551
entry supersedes it and no current code or deployment builds it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PyRIT's default objective scorer is chosen by F1, but
ScorerInitializer._tag_best_objectiveonly ranksregistry entries with complete coverage counts (
num_input_responses). Most objective entries predate thatfield, so the only eligible candidate was
acs_with_refusal(F1 0.366, below the always-true baseline of0.652), and it became the default. This PR re-measures the strongest objective scorers with current code so
selection can see them. With it, the default changes from
acs_with_refusaltoinverted_refusal(F1 0.885).responses determined). Identities and eval hashes are unchanged; each entry is replaced in place by
replace_evaluation_results(it moves to the end of the file).19c69859…) that isinverted_refusalrecorded with the unresolved placeholder{AZURE_OPENAI_GPT5_4_UNDERLYING_MODEL}as its underlying model name, which gave it an eval hash noscorer produces. That entry recorded F1 0.896 with complete coverage; the re-measured 0.885 is within
noise of it, not a regression.
inverted_refusal(gpt-5.4)fallback(gpt-4o)main(gpt-4o-unsafe)acs_with_refusal(default before this PR; no longer selected)Constant-guess baselines on this set: accuracy 0.516, F1 0.652. The before/after differences for
inverted_refusalandfallbackare within one combined standard error (about 0.022), so this is are-measurement, not an accuracy improvement; the user-visible change is which scorer is selected by default.
mainis the inverted refusal scorer onobjective_scorer_chat, which previously had no metrics; it trailsinverted_refusalby 0.025 accuracy, about 1.5 combined standard errors, so its lower ranking is suggestiverather than conclusive.
Each entry was produced the way
build_scripts/evaluate_scorers.pydoes it: the registeredScorerInitializerinstance (inverted_refusal,fallback,main) run withevaluate_async(num_scorer_trials=3)against its own objective gold mapping, using Azure OpenAI judges.🤖 Generated with Claude Code