Skip to content

Test oracle approach which directly compares Isabelle (SML backend) to its extracted Python - #4

Merged
nikolaushuber merged 16 commits into
mainfrom
isabelle-python-direct-compare
Nov 2, 2026
Merged

nikolaushuber merged 16 commits into
mainfrom
isabelle-python-direct-compare

Conversation

@dorchard

@dorchard dorchard commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

This setup allows tests cases to be written in Isabelle that are then extracted to Python and compared directly+autoamtically. At build time, python_check evaluates the Isabelle using its SML code generator, then exports a Python function checking that the generated code computes the same value. There is a pytest hook that then runs every tests. This avoids the need for more hand written tests so can scale perhaps slightly more easily? I then got Claude to generate a lot of tests in Oracle_*.thy. There appears to be various issues thrown up! So this currently fails. Maybe we could fix them on this PR to get the CI green.

Run with:

make build-test-oracle
make test-oracle

I can try to tackle these...

$ make test-oracle

FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:nat_of_integer_neg] - Failed: nat_of_integer (- 5)
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:l_upt_long] - RecursionError: maximum recursion depth exceeded
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:s_min] - TypeError: '<' not supported between instances of 'seta' and 'seta'
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:s_max_sets] - TypeError: '>' not supported between instances of 'seta' and 'seta'
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:str_eq_chain] - Failed: (STR ''a'' = STR ''b'') = (STR ''c'' = STR ''d'')
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:str_eq_false] - Failed: (STR ''a'' = STR ''b'') = False
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:str_le_eq] - Failed: (STR ''b'' \<le> STR ''a'') = (STR ''a'' = STR ''b'')
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:str_bool_eq] - Failed: (STR ''x'' = STR ''x'') = (1 = 1)
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:k_async] -   File "/Users/dorchard/Documents/isabelle-python-codegen/test/oracle/python_out/Python_Test_Oracle.Oracle_Cas...
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:k_vars] -   File "/Users/dorchard/Documents/isabelle-python-codegen/test/oracle/python_out/Python_Test_Oracle.Oracle_Cas...
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases:sort_str] - AttributeError: partially initialized module 'String' from '/Users/dorchard/Documents/isabelle-python-codegen/...
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:nat_of_integer_neg] - Failed: nat_of_integer (- 5)
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:l_upt_long] - RecursionError: maximum recursion depth exceeded
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:s_min] - TypeError: '<' not supported between instances of 'seta' and 'seta'
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:s_max_sets] - TypeError: '>' not supported between instances of 'seta' and 'seta'
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:str_eq_chain] - Failed: (STR ''a'' = STR ''b'') = (STR ''c'' = STR ''d'')
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:str_eq_false] - Failed: (STR ''a'' = STR ''b'') = False
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:str_le_eq] - Failed: (STR ''b'' \<le> STR ''a'') = (STR ''a'' = STR ''b'')
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:str_bool_eq] - Failed: (STR ''x'' = STR ''x'') = (1 = 1)
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:k_async] -   File "/Users/dorchard/Documents/isabelle-python-codegen/test/oracle/python_out/Python_Test_Oracle.Oracle_Cas...
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:k_vars] -   File "/Users/dorchard/Documents/isabelle-python-codegen/test/oracle/python_out/Python_Test_Oracle.Oracle_Cas...
FAILED test/oracle/run_test.py::test_agrees_with_isabelle[Python_Test_Oracle.Oracle_Cases_TC:sort_str] - AttributeError: partially initialized module 'String' from '/Users/dorchard/Documents/isabelle-python-codegen/...

@rolyp

rolyp commented Oct 23, 2026

Copy link
Copy Markdown

@dorchard Nice!

@dorchard

Copy link
Copy Markdown
Contributor Author

Made a few process improvements (mostly better error reporting) and added to the CI. I can make a separate PR off this one with the fixes.

@dorchard dorchard changed the title Test oracle approach which directly compares Isabelle to its extracted Python Test oracle approach which directly compares Isabelle (SML backend) to its extracted Python Oct 23, 2026
rolyp pushed a commit that referenced this pull request Oct 24, 2026
@nikolaushuber

Copy link
Copy Markdown
Collaborator

CI/oracle is supposed to fail without the fixes in #6

@nikolaushuber
nikolaushuber merged commit c1af22c into main Nov 2, 2026
3 of 4 checks passed
@nikolaushuber
nikolaushuber deleted the isabelle-python-direct-compare branch November 2, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants