[Lanre Shittu (Shizoqua)]

Summary

Fixes #398.

When a SearchQA item's answers field was a single string instead of a list, evaluate() and the EM, F1 and sub_em helpers iterated its characters. The correct answer Paris scored EM 0, while P or s scored EM 1 and F1 1. A single string gold answer is now treated as a one-item list.

User impact: hard is EM and soft is F1, so such items gave an inverted training reward (correct answers penalized, one-letter answers rewarded) and wrong reported scores.

Proof: the regression tests fail on upstream 45f65ca and pass with this change.

Root cause

evaluate() in skillopt/envs/searchqa/evaluator.py passed gold_answers straight to exact_match(), f1_score() and sub_em(), which loop over it. Iterating a str yields characters. materialize_searchqa.py writes lists, but custom data or a hand-written split can store "answers": "Paris", and nothing normalized it. The DocVQA evaluator already accepts a single string gold answer.

Changes

• evaluate() wraps a string gold_answers in a list before scoring. This is the entry point both rollout paths in rollout.py call, so they are both covered. List gold answers are unchanged.
• The returned gold_answers is the normalized list, so result rows show ["Paris"] instead of the bare string.
• Regression tests added to tests/test_searchqa_evaluator.py.
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.searchqa.evaluator import evaluate

for response in ["<answer>P</answer>", "<answer>s</answer>", "<answer>Paris</answer>", "<answer>London</answer>"]:
    result = evaluate(response, "Paris")
    print(response, result["em"], result["f1"], result["sub_em"])

Before:

<answer>P</answer> 1.0 1.0 1.0
<answer>s</answer> 1.0 1.0 1.0
<answer>Paris</answer> 0.0 0.0 1.0
<answer>London</answer> 0.0 0.0 0.0

After: Paris scores 1.0 1.0 1.0 and the other three score 0.0 0.0 0.0.

Testing

• python -m pytest tests/test_searchqa_evaluator.py -q: on upstream 45f65ca, 3 failed (assert 1.0 == 0.0 and assert 0.0 == 1.0) and 6 passed; with the fix, 9 passed.
• python -m pytest tests/test_searchqa_evaluator.py tests/test_searchqa_rollout_failfast.py tests/test_materialize_searchqa.py tests/test_qwen_backend.py -q: 43 passed.
• python -m pytest -q: 2219 passed, 12 skipped, 4 xfailed.
• python -m ruff check and python -m ruff format --check on the changed files: clean.
• python -m mkdocs build --strict: passed.

Risk and compatibility

Only calls that pass a string as gold_answers change, and those were scored incorrectly before. List inputs are scored exactly as before. No backend, config key, file format or CLI change.