[Lanre Shittu (Shizoqua)]

Summary

Fixes #400.

The OfficeQA loader read the gold answer with str(row.get("ground_truth") or row.get("answer") or ""). In a JSON split, a numeric gold answer of 0 or 0.0 is falsy, so it was skipped: the item loaded with an empty ground_truth and no answers, and a model answering 0 scored EM 0 and F1 0. The loader now takes the first field that is present and not blank.

User impact: OfficeQA answers are mostly numbers from Treasury tables, so 0 is a plausible gold value. Those items gave zero reward for the correct answer and counted as misses in reported scores.

Proof: the regression tests fail on upstream 45f65ca and pass with this change.

Root cause

_normalize_row() in skillopt/envs/officeqa/dataloader.py chained or across ground_truth and answer. or treats 0 and 0.0 as missing. CSV splits are not affected because CSV values are strings, but JSON splits keep numeric types.

Changes

• New _first_text(row, *keys) helper returns the first value that is not None and not blank, as stripped text. _normalize_row() uses it for the gold answer.
• String values, the answer fallback, and items with no gold answer behave as before. A whitespace-only ground_truth now also falls back to answer instead of producing an empty gold.
• New tests/test_officeqa_dataloader.py, loading real JSON split files through OfficeQADataLoader.load_split_items().
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

import json, pathlib, tempfile
from skillopt.envs.officeqa.dataloader import OfficeQADataLoader
from skillopt.envs.officeqa.evaluator import evaluate

split = pathlib.Path(tempfile.mkdtemp()) / "val"
split.mkdir()
rows = [{"uid": "q1", "question": "How many bonds were issued?", "ground_truth": 0}]
(split / "items.json").write_text(json.dumps(rows))

for item in OfficeQADataLoader().load_split_items(str(split)):
    print(item["uid"], repr(item["ground_truth"]), item["answers"], evaluate("0", item["ground_truth"])["em"])

Before: q1 '' [] 0.0

After: q1 '0' ['0'] 1.0

Testing

• python -m pytest tests/test_officeqa_dataloader.py -q: on upstream 45f65ca, 3 failed (assert '' == '0' and similar) and 2 passed; with the fix, 5 passed.
• All OfficeQA-related tests (tests/test_officeqa_*.py and tests/test_benchmark_task_output_paths.py): 112 passed.
• python -m pytest -q: 2221 passed, 12 skipped, 4 xfailed.
• python -m ruff check on the changed files: one F401 for an unused import os that is unchanged from main; the new test file is clean.
• python -m mkdocs build --strict: passed.

Risk and compatibility

Only JSON items whose gold answer is numeric zero, or whose ground_truth is whitespace-only, load differently. No backend, config key, file format or CLI change. A gold value of 0.0 now loads as "0.0"; whether 0 and 0.0 should match is left to the evaluator's existing normalization.