Repository navigation
Conversation
The equity and gwr risk-aversion tests fail intermittently in GitHub Actions: the same commit passes on one run and fails on the next, and origin/main fails the same way. All thirty failures sit downstream of dscim.utils.utils.c_equivalence, the CRRA certainty equivalent computed from the float32 CE zarr stores. GitHub's hosted runners are a shared pool of hosts with and without AVX512, and numpy 1.26.4 dispatches np.power to the Intel SVML kernels bundled in its wheel only when AVX512_SKX is present. The last-bit difference that produces becomes a 1e-4 to 1e-2 relative difference in damage_function_points once the two certainty equivalents are subtracted, which is past the tolerances of the points, coefficients and fit tests. Fifteen diagnostic jobs on the fork show the two outcomes are each deterministic: every job with numpy's AVX512 targets enabled (Intel Xeon 8573C, 6973P-C, 6975P-C, AMD EPYC 9V74, 9V45) gives one wrong fingerprint and fails 27 of 35 parametrisations, with one thread or many; every job on the AVX2 path (AMD EPYC 7763, or any host with NPY_DISABLE_CPU_FEATURES set) reproduces the reference fixtures bit for bit and passes 35 of 35. A local sweep in Docker over BLAS and OpenMP thread counts, PYTHONHASHSEED, the dask scheduler and the AVX2 and FMA3 dispatch was bit-identical in all 21 configurations, so none of those is the variable. Set NPY_DISABLE_CPU_FEATURES on the test job so numpy uses the AVX2 loops on every host. numpy accepts the variable silently on hosts that lack the listed features. Pin both jobs to ubuntu-24.04 so the runner image does not change between runs on its own. No test, tolerance or library code changes.
NPY_DISABLE_CPU_FEATURES names dispatch targets, and numpy disables exactly the names given, so the list only covers the target names of the numpy build it was written for. numpy 1.26.4 and the 2.0 to 2.2 wheels dispatch the same sixteen names; from numpy 2.4.0 the x86 targets are X86_V3, X86_V4, AVX512_ICL and AVX512_SPR, and np.power's SVML kernel is built for X86_V4. Under that naming the present list disables only AVX512_ICL and numpy raises an ImportWarning for the other names, so the fix would stop working on AVX512 hosts without any test failing in a visible way. Add a step before pytest that reads numpy's enabled dispatch targets, through numpy._core on 2.x and numpy.core on 1.x, prints the host CPU and the target lists, and exits non-zero when any AVX512_* target or X86_V4 is enabled. The message names NPY_DISABLE_CPU_FEATURES and the list the installed numpy needs. Verified in Docker on numpy 1.26.4, 2.4.0 and 2.5.4, including the failure path with an AVX512-class target forced on.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #488 +/- ##
=======================================
Coverage 68.94% 68.94%
=======================================
Files 15 15
Lines 1755 1755
=======================================
Hits 1210 1210
Misses 545 545 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five tests have been failing intermittently on
main:tests/test_equity.py::test_equity_pointstests/test_equity.py::test_equity_coefficientstests/test_equity.py::test_equity_fittests/test_risk_aversion.py::test_risk_aversion_coefficientstests/test_risk_aversion.py::test_risk_aversion_fitThe failures appear to come from how NumPy chooses CPU-specific implementations at runtime. On AVX512 hosts,
np.poweruses SVML kernels built forAVX512_SKX; on other hosts, it uses the generic AVX2 loops. These implementations can differ in the last bit, as discussed in numpy/numpy#23523.That small difference matters here because all five tests depend on the CRRA certainty equivalent, and damages are calculated as the difference between two nearly equal certainty equivalents. This amplifies the difference to roughly 1e-4 to 1e-2 relative. The fixtures were generated using the AVX2 path, and nine jobs on AVX512 hosts produced the same failing result, suggesting two deterministic outcomes rather than random noise. NumPy considers this a speed-versus-consistency tradeoff and accepts differences of up to 4 ULP from the reference results.
This change is limited to the workflow and a CHANGELOG entry; it does not modify the tests or library code. It:
NPY_DISABLE_CPU_FEATURESon the test job to disable the eight AVX512 targets dispatched by the pinned NumPy 1.26.4.ubuntu-24.04instead ofubuntu-latest.The check is there to catch future changes to NumPy's target names. The disabled features are an enumeration rather than a hierarchy, and NumPy 2.4 renames the targets. As described in numpy/numpy#33016, the old names are ignored with an
ImportWarningthat Python hides by default. When we upgrade, the list will need to change toX86_V4 AVX512_ICL AVX512_SPR. We should not addX86_V3, since that is the AVX2-level target used by the fixtures. The Dependabot branch for NumPy 2.4.0 is already open. The target names are documented in the NumPy SIMD build options.The workflow passed three consecutive runs, with no change in pytest duration.
cc @brews