[Lanre Shittu (Shizoqua)]

Summary

Fixes #399.

DocVQA's extract_answer() rejected the final answer when the response mentioned <answer> again after it, for example <answer>42</answer> (formatted with the <answer> tag as asked). It then scored the last line of text instead, so a correct answer got ANLS 0. The last </answer> is now paired with the last <answer> before it.

User impact: ANLS is the DocVQA soft score and hard is ANLS >= 0.999, so a correct answer followed by a stray tag got zero training reward and counted as a miss in reported scores.

Proof: the regression tests fail on upstream 45f65ca and pass with this change.

Root cause

extract_answer() in skillopt/envs/docvqa/evaluator.py took rfind("<answer>") and rfind("</answer>") independently and used them only when the closing tag came after the opening one. A stray <answer> after the final </answer> made the last opening tag come after the last closing tag, so the valid pair was discarded and the last-line fallback ran.

Changes

• extract_answer() finds the last </answer>, then the last <answer> before it. This is still a linear scan over the text.
• Responses without a stray tag are extracted as before. Nested input such as <answer>x<answer>y</answer> still returns y, and the last-line fallback is unchanged when there is no closing tag.
• Regression tests added to tests/test_docvqa_evaluator.py, including the nested case as a guard.
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.docvqa.evaluator import evaluate, extract_answer

for text in [
    "<answer>42</answer> (formatted with the <answer> tag as asked)",
    "The total is <answer>42</answer>\nNote: only one <answer> block.",
    "<answer>42</answer>\n<answer>",
]:
    print(repr(extract_answer(text)), evaluate(text, ["42"])["anls"])

Before:

'<answer>42</answer> (formatted with the <answer> tag as asked)' 0.0
'Note: only one <answer> block.' 0.0
'<answer>' 0.0

After: '42' 1.0 for all three.

Testing

• python -m pytest tests/test_docvqa_evaluator.py -q: on upstream 45f65ca, 3 failed (assert '<answer>' == '42' and similar) and 33 passed; with the fix, 36 passed.
• python -m pytest tests/test_docvqa_evaluator.py tests/test_benchmark_task_output_paths.py -q: 52 passed.
• python -m pytest -q: 2220 passed, 12 skipped, 4 xfailed.
• python -m ruff check and python -m ruff format --check on the changed files: clean.
• python -m mkdocs build --strict: passed.
• Adversarial input (200,000 repeated tags) is extracted in a few milliseconds.

Risk and compatibility

Only responses with an <answer> after the last </answer> change outcome, and those were scored from the wrong text before. No backend, config key, file format or CLI change.