[eliotwrobson]

Summary

query_setup asked the sandbox for dill-encoded values after student code had run. Setup and student code share a subprocess, so student code could forge the data that the grader then unpickled, and the grader process appears to run as root. Experiments confirmed student code can reach the runner through traceback frames with no imports, and that an ordinary global assignment can end up in a setup function's pickle and run when it's loaded.

The grader now only unpickles data captured before student code starts:

• Values JSON represents exactly (plain data, numeric NumPy arrays and scalars, DataFrames) stay in the sandbox and are sent as JSON only when queried, so large values add nothing to start time.
• Other values, such as functions, classes, tuples and sets, are dill-encoded once after setup code finishes. The runner waits for an acknowledgement before starting student code, and query_setup reads these from the stored copy. Function payloads leave out globals the function doesn't use.
• Nothing sent after student code starts is ever unpickled.
• A value that can't be serialized raises a TypeError naming the variable. Other setup variables still work.

This PR also adds plot-serializer==0.2.3 to the full image, and corrects the docs to say plot grading needs MatplotlibSerializer rather than a plain Figure.

Loading data.json in the grader images

PrairieLearn writes /grade/data/data.json, but the data_json fixture only read data.json next to the test file, and nothing put it there. Any test using data_json["params"] failed with 'NoneType' object is not subscriptable, so correct submissions scored 0.

• run.sh now passes --data-json-path=/grade/data/data.json to pytest. Without that option, or if that file doesn't exist, the fixture reads data.json next to the test file, which local runs and the scenario tests rely on.
• data.json can contain correct_answers, so run.sh makes the data directory and file root-owned with owner-only modes before student code runs. If that fails, grading stops with a failure result instead of running student code. Root ownership is needed because the student uid (1000) can match the host owner of mounted files. Owner-only modes are needed because student code keeps root's supplementary groups.
• The fixture still returns None when data.json is missing, but now raises a clear error when it can't be parsed or read.
• run.sh takes the job directory as an optional argument, defaulting to /grade, so tests/test_run_script.py can run the real script. It no longer exports JOB_DIR or MERGE_DIR.

Release together: the image's run.sh and package now depend on each other. This run.sh with package 1.1.0 fails on the unknown option and writes a failure result for every submission.

Behaviour changes

• JSON values are read when queried, so student code that gets around the sandbox could change them, including expected answers. It can't run code in the grader this way. This is no worse than main.
• Dill values are as they were when setup code finished. A setup function's globals hold instructor values, not anything the student rebound.
• Values JSON can't represent are still encoded for each sandbox, even if never queried. A large one, such as a trained model, still slows each start.
• query_setup now takes query_timeout, defaulting to 1 second, because JSON values travel over the socket.

Known gap

The dill snapshot protects against code in the same subprocess. Each test starts a new subprocess under the same ag user, and cleanup only terminates that one process. From reading the code, a background process left by student code in an earlier test could tamper with a later subprocess before its snapshot is taken. This wasn't tried, and closing it would need per-test process cleanup or separate users.

Testing

Run on Python 3.14:

• uv run pytest tests -p no:prairielearn-grader: 104 passed on the final code, with one earlier test_initialization_timeout failure described below
• mypy src/: clean
• ruff check: two new PLC0415 lazy-import findings in json_utils.py, the same pattern to_json uses; otherwise no change from main
• New test_query_setup_snapshot scenario: student code forges malicious pickles and JSON responses, overwrites setup values and rebinds a global. An earlier version of it failed 4 of 5 tests on main.
• A sandbox with a 128 MB array and a 2-million-item list in setup code started in about 0.45s, including creating them.
• Full image built on Python 3.14 Alpine, using the published 1.1.0 package with this branch's source mounted over it. With privileges dropped to ag, the new scenario scored 5/5 and test_matplotlib_plots scored 12/12.
• Lite image with this branch's source mounted over the package, real run.sh, a /grade layout copied onto the container's native filesystem: the converted randomFunctionName question scored 1.0 for a correct submission and 0.6 for a wrong quest value, with files starting out owned by root or by uid 1000.
• In the same setup, a submission running as ag got Permission denied on /grade/data/data.json and on listing /grade/data, while the grader still loaded the file.
• Docker Desktop's macOS bind mount did not enforce these modes (ag could read the file), which appears to be specific to that mount. A Linux host bind mount, which is what production uses, was not tested.

test_matplotlib_plots failed on the first run in fresh local environments, with and without this change, while matplotlib was building its font cache. It looks like a first-run timeout, but that isn't confirmed.

test_initialization_timeout failed once in an earlier full suite run. Its failure details weren't saved. It then passed 8 times alone, and 5 further full runs each on this branch and main had no failures, so the cause is unconfirmed and it may be a rare timing issue.

Follow-ups

• ag can still read other grading material: /grade/run/test_student.py, and everything from /grade/tests such as setup_code.py. It can also reach params in sandbox memory through gc, so values in params aren't hidden from student code.
• drop_privileges doesn't call setgroups, so student code keeps root's supplementary groups (0, 1, 2, 3, 4, 6, 10, 11, 20, 26, 27).
• If the host user that owns the job files has uid 1000, ag also owns the moved test files and could likely modify them, including setup_code.py, which is read again for each sandbox. This comes from reading the code and wasn't tried.
• Returning a plain Figure from student code is still unsupported. From reading the code, it fails in the handler's json.dumps and closes the connection, which likely gives the author an obscure error.
• A package release and image build are needed before PrairieLearn questions get these changes.

🤖 Generated with Claude Code