[Aditya-Pathanjali]

Fixes #702

purify visits the surfaces in a new random order on every sweep and stopped at the first sweep whose total impurity did not shrink compared to the previous sweep. With a random order that total is not monotone, and empty cells make convergence slow, so tensors with empty cells were often left impure, silently. With is_randomized=False the same inputs purify to about 1e-16.

Change

When the order is random, a sweep that fails to shrink the total now ends the loop only once the total has dropped to floating point noise, defined as 1e-12 of the first sweep. Otherwise the loop continues until 20 sweeps in a row fail to set a new minimum, which also guarantees that it terminates.

• The fixed-order path (is_randomized=False) keeps the old exit condition and is unchanged.
• The multiclass loop (PurifyNormalizedMulticlass) had the same exit condition and gets the same change. I confirmed the multiclass case fails on the released library too.
• The logic is in one helper, IsPurifySweepingDone, used by both loops.

I picked the exit rule by simulating the sweep loop in Python on 5x6, 3x3x3, 4x5x6, 8x8x8, 32x32 and 10x10 tensors:

• The current rule fails 91 to 96 of 100 on the tensors with empty cells.
• Switching to a fixed order after the first stall still failed 38 to 61 of 100.
• A patience-only rule (20 sweeps without a new minimum) fixes everything but runs about 2.4 to 2.7x more sweeps on inputs that already worked.
• The chosen rule fixes all of them and takes exactly the same number of sweeps as today on inputs that already worked.

Evidence

Before and after on 100 random tensors each, tolerance=0, is_randomized=True, worst weighted row or column sum:

Shape (empty cells)	Not pure before	Not pure after	Worst sum after
5x6 (60%)	92 / 100 (worst 0.88)	0 / 100	9e-12
3x3x3 (30%)	94 / 100 (worst 0.64)	0 / 100	5e-12
32x32 (0%)	0 / 100	0 / 100	8.2e-15 (identical)

The "before" numbers are from the released 0.7.8 wheel. The "after" numbers are from a build of this branch.

Tests added:

• PurifyTest.cpp: two cases (single class and multiclass) with 50 pseudo random 5x6 tensors each, 60% empty cells, zero tolerance. They check the weighted row and column sums directly because MeasureImpurity returns NaN for a row or column with no weight. Both fail on the original Purify.cpp (50 and 126 failed checks) and pass with this change.
• test_purify.py: two cases (3 tensor shapes, and a multiclass shape) with randomized order and empty cells. Both fail against the released 0.7.8 wheel and pass with this change.

Checks run, Windows 11:

• Full native test suite (shared/libebm/tests, 348 tests): all pass.
• python/interpret-core/tests/utils: 225 passed, 9 skipped. I excluded test_measure_interactions.py, which needs the dash package.
• An end-to-end ExplainableBoostingClassifier fit with purify_result enabled gives the same accuracy and the same purity on the original and the fixed library, in a similar time (5.4 s and 5.2 s).
• clang-format on the changed C++ files and ruff format --check on the changed Python file are clean. The clang-format I had is version 12, not 16.

Limits

• This machine has Visual Studio 2019 Build Tools, and the project targets v143. Its compiler also rejects an unrelated overload in Objective.hpp. I built the library and the tests with clang (via the ziglang package) for x86_64 Windows, CPU compute unit only, with the same flags as build.sh (-std=c++11 -O3) and libebm_exports.def. The AVX2 and AVX-512 units, the MSVC build, Linux, macOS, and the R package were not built or run here.
• The ebm interaction tensors in my end-to-end check were already pure, so that check shows no change for them, not that the fix helps in training. purify_result is off by default, as the issue notes.
• 20 sweeps and 1e-12 are chosen from simulation. I can change either if you prefer other values.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.77%. Comparing base (003136e) to head (47f4b9b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #708      +/-   ##
==========================================
- Coverage   67.80%   67.77%   -0.04%     
==========================================
  Files          77       77              
  Lines       11893    11893              
==========================================
- Hits         8064     8060       -4     
- Misses       3829     3833       +4     

Flag	Coverage Δ
bdist_linux_311_python	67.54% <ø> (ø)
bdist_linux_312_python	?
bdist_linux_313_python	?
bdist_linux_314_python	67.45% <ø> (ø)
bdist_linuxarm_311_python	67.52% <ø> (-0.04%)	⬇️
bdist_linuxarm_312_python	?
bdist_linuxarm_313_python	?
bdist_linuxarm_314_python	67.47% <ø> (ø)
bdist_mac_311_python	67.67% <ø> (-0.03%)	⬇️
bdist_mac_312_python	?
bdist_mac_313_python	?
bdist_mac_314_python	67.61% <ø> (-0.02%)	⬇️
bdist_win_311_python	67.72% <ø> (ø)
bdist_win_312_python	?
bdist_win_313_python	?
bdist_win_314_python	67.59% <ø> (-0.04%)	⬇️
sdist_linux_311_python	?
sdist_linux_312_python	?
sdist_linux_313_python	?
sdist_linux_314_python	?
sdist_linuxarm_311_python	?
sdist_linuxarm_312_python	?
sdist_linuxarm_313_python	?
sdist_linuxarm_314_python	?
sdist_mac_311_python	?
sdist_mac_312_python	?
sdist_mac_313_python	?
sdist_mac_314_python	?
sdist_win_311_python	?
sdist_win_312_python	?
sdist_win_313_python	?
sdist_win_314_python	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.