Repository navigation
Description
Overview
factory._use_interferometer_numba routes a NumPy single-mapper interferometer inversion to the numba direct_conv curvature route only when mean non-zeros per source column ≤ interferometer_numba_nnz_per_source_max (default 60, calibrated on the 125×125 reference mask). On the 512×512 Aris mask (91,507 masked pixels, 682×682 extent) the measured crossover is ~140–150 nnz/column: at 2,577 vertices (106 nnz/col) the gate sends the fit to the FFT route at 44.1 s/call against 34.2 s for direct_conv. FFT cost scales ~linearly with source pixels and with the FFT extent; direct_conv scales with total non-zeros × kernel work and is ~flat in source pixels. A fixed nnz/column constant cannot be right for both masks, so the gate becomes a cost model over quantities known at routing time.
Plan
- Replace the fixed nnz/column gate with a two-term cost model: FFT route ∝ S × (padded extent FFT cost, batched 128 columns); direct_conv ∝ total nnz × per-nonzero kernel work; route to whichever predicts cheaper.
- Calibrate the two constants on the existing measurements (125×125: crossover ~60 Delaunay / ~77 rectangular, autolens_profiling#226; 512×512: FFT 28.3/44.1/97.3 s and direct_conv 38.4/34.2/41.2 s at 1,459/2,577/5,746 vertices).
- Keep a user override: an explicitly set
interferometer_numba_nnz_per_source_maxstill wins, 0 remains the kill switch; add a force-route setting for benchmarking. - Add routing tests reproducing both masks' geometries that assert the faster route at every measured vertex count, and keep the DelaunayNN numba-routing pins (PyAutoArray#625) green.
- Phase 2 (autolens_profiling, after the library PR): re-measure on both masks with the new default and record the routing table.
Tier: glance — merge mode: auto-merge on green if the Witness passes, post-merge summary
Detailed implementation plan
Affected Repositories
- PyAutoArray (primary, phase 1)
- autolens_profiling (phase 2, re-measurement record)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| array/PyAutoArray | main (a84f724, includes DelaunayNN #624/#625) | clean |
| lens/autolens_profiling | main | 1 uncommitted file in canonical checkout (Heart yellow; not ours) |
Suggested branch: feature/interferometer-numba-route-gate
Implementation Steps
autoarray/inversion/inversion/interferometer_numba/inversion_interferometer_numba_util.py— besidennz_per_source_column_from, addroute_cost_from(mapper, extent_shape, batch_size)returning(t_fft, t_direct_conv)estimates:t_fft = c_fft * S * ceil-batched extent FFT work (extent_y*extent_x*log2(extent_y*extent_x)),t_dc = c_dc * nnz_total * kernel_work(derive kernel work from the direct_conv implementation: per-nonzero it touches the (2ny,2nx) operator preload rows for the column's other non-zeros; confirm by readingcurvature_direct_conv). Fitc_fft,c_dcby least squares to the six 512×512 points and the 125×125 crossovers; document the fit residuals in the docstring.autoarray/inversion/inversion/factory.py_use_interferometer_numba(~L242–320) — replace thennz_per_source_column > nnz_maxcomparison with: explicit user value set → old behaviour (honoursautolens_profilingA/B scripts); default → cost-model comparison. The extent shape comes fromdataset.sparse_operator/ maskshape_native_masked_pixels; pass what the factory already has (it may need the dataset, not just the mapper — adjust the signature and its two callers/tests).autoarray/settings.py— makeinterferometer_numba_nnz_per_source_maxdefault toNonemeaning "cost model" in the packaged config, keep0as kill switch and a positive value as a hard gate; addinterferometer_route: Optional[str]("numba"/"fft") force override for benchmarking. Update the config docstrings and theconfig/default yaml.- Tests:
test_autoarray/inversion/inversion/test_factory.py— synthetic mappers with the 125×125 geometry (5,496 px, extent ~170², nnz/col 50 → numba, 80 → FFT for Delaunay; ~77 crossover rectangular) and the 512×512 geometry (91,507 px, 682² extent, S=1459/188 nnz → FFT, 2577/106 → numba, 5746/48 → numba); assert the chosen class. Keeptest__inversion_imaging__delaunay_nn__numba_*and test: pin DelaunayNN routing to the numba likelihood paths (#623 phase 2) #625's routing pins green. Settings tests for the override semantics. - Witness: with the scratchpad
aris_evidence/step8/bench.py --routeon the 512 cache at s=8/6/4 and the 125×125 inputs viastep7/bench.py, confirm the default picks the faster route at every vertex count; record the table in the PR body. - Phase 2 (workspace,
start_workspaceafter the library PR merges): add the re-measurement record toautolens_profiling(the campaign's results directory, same format as docs: refactor docstrings for autoarray/mask package #226) and reference this issue.
Key Files
autoarray/inversion/inversion/factory.py—_use_interferometer_numbaautoarray/inversion/inversion/interferometer_numba/inversion_interferometer_numba_util.py—nnz_per_source_column_from,curvature_direct_convautoarray/settings.py+autoarray/config/— gate setting and defaultstest_autoarray/inversion/inversion/test_factory.py— routing tests
Ordering
Runs after the sparse data-vector task (same repo claim; it also lowers memory for the re-measurement). DelaunayNN routing (#624/#625) merged 2026-10-15, so no factory.py conflict remains.
Evidence
PROGRESS4 steps 8.3a/b/b'/profile; measured s/call, 1 thread. Evidence copied to this session's scratchpad aris_evidence/ (temporary).
Heart RED development override
Heart was RED at launch (2026-10-15 evening): Heart: RED — Release RED; monitoring RED · 10/100 · 415 unresolved; [red] release validation FAILED (stage integrate); [red] PyAutoHands/workflows/Python Version Matrix: failure; [red] PyAutoHeart/workflows/Release Integrate: failure; [red] PyAutoMind/workflows/Lifecycle Drift: failure; [red] PyAutoArray/Tests: red; [red] autofit_workspace_test/Smoke Tests: red; [red] autogalaxy_workspace_test/Smoke Tests: red; [red] autolens_workspace_test/Smoke Tests: red; [red] Release validation: NOT release_ready — v2026.10.15.1.dev81701; [red] Worktree drift: 5 orphan / 0 missing / 4 dirty; [red] autolens_inference: red; [red] autolens_profiling: red; [red] euclid_strong_lens_modeling_pipeline: red.
The live human authorized the development-only override in session 3e3dea38 (AskUserQuestion answer: "Yes, override for development (Recommended)") for the three PyAutoArray prompts filed from the Aris interferometer profiling. Permitted: commit, push, PR-open behind the autonomous-ship gate (tests, smoke, review must still pass). Merge stays a separate human act. Heart remains RED for release purposes. Launched --auto.
Original Prompt
Click to expand starting prompt
Interferometer numba route gate should scale with mask size
Type: feature
Target: PyAutoArray
Repos:
- PyAutoArray
- autolens_profiling
Difficulty: large
Autonomy: supervised
Priority: normal
Memory: wiki/galaxies/sources/massive-ellipticals.md; wiki/lensing/sources/dark-matter-substructure.md; wiki/galaxies/sources/cosmos-survey.md
Status: formalised
Consequence: glance
Witness: on both the Aris 125x125 (5,496 px) and 512x512 (91,507 px) masks, the default routing picks the faster of numba direct_conv vs NumPy FFT curvature at every tested vertex count (1.4k-5.7k), asserted by a routing test
Review-minutes: 3
Unattended: ready
The factory (autoarray/inversion/inversion/factory.py ~L245) routes a NumPy single-mapper
interferometer inversion to the numba direct_conv curvature route only when mean non-zeros per
source column <= interferometer_numba_nnz_per_source_max (default 60, measured on one reference
geometry). On a 512x512 mask (91,507 masked pixels) the measured crossover is ~140-150 nnz/column:
at 2,577 vertices (106 nnz/col) the default sends numba to the FFT route at 44.1 s vs 34.2 s for
direct_conv (1.29x slower). Measured (s/call, 1 thread):
| vertices (nnz/col) | FFT route | direct_conv |
|---|---|---|
| 1,459 (188) | 28.3 | 38.4 |
| 2,577 (106) | 44.1 | 34.2 |
| 5,746 (48) | 97.3 | 41.2 |
FFT cost scales ~linearly with source pixels (blocks of 128 FFTs over the 682x682 extent) and with
extent size; direct_conv scales with total non-zeros x kernel work. A fixed nnz/column constant
cannot be right for both small and large masks.
Do: replace the fixed gate with a cost model using quantities known at routing time (source pixels
S, total nnz, extent / FFT grid shape), calibrated on the existing measurements (old 125x125 mask
crossover ~60 Delaunay / ~77 rectangular, autolens_profiling#226; this 512x512 data), keeping a
user override. Add tests for the routing decision. Re-measure on both masks.
Evidence: /tmp/claude-1000/-home-jammy-Code-PyAutoLabs/50c8c6cd-6ac2-47c2-9290-7ee73f9c4315/scratchpad/aris/PROGRESS3.md, PROGRESS4.md, step7/, step8/. PyAutoArray ccddfba.
Activity
Jammy2211 commentedon Oct 15, 2026
Campaign membership: this task is logged and monitored under the PyAutoPulse interferometer campaign as task
interferometer_large_mask_aris— PyAutoLabs/PyAutoPulse#62 (ledger rows queued in Mind behind the PyAutoPulse claim; the issue holds the campaign intent and baseline numbers until they land). Record the before/after witness table there at close-out.Jammy2211 commentedon Oct 15, 2026
Library PR Created
PR: #632 (
pending-release,decision-taken). This run stopped at PR-open: merging under the Heart RED override is a separate human/prm.What changed: the numba
direct_convvs NumPy FFT interferometer curvature route used to be picked by a fixed 60 non-zeros-per-column gate. It is now picked by a two-constant cost model:t_fft = c_fft · S · [rfft2+irfft2 work on the (2ny,2nx) padded extent], where the transform work uses pocketfft's own factorisation-awarecost_guessestimate.t_dc = c_dc · nnz · ny · nx.c_fft = 7.62e-10andc_dc = 1.77e-09, fitted on 9 paired timings (6 bake-off cells from autolens_profiling#226 plus the 3 Aris 512 points). The faster route is predicted correctly at all 9; ratio residuals are 0.68-1.47.Settings:
interferometer_numba_nnz_per_source_maxnow defaults tonull, which means the cost model. A positive value is still a hard gate and0still disables numba. Newinterferometer_route: "numba" | "fft" | nullforces a route.Witness: passed.
Gate:
release validation FAILED (stage integrate)). We shipped under the development-only override recorded above.Workspace impact: option (iii), no workspace change needed.
interferometer_numba_nnz_per_source_maxdoes not appear in autolens_workspace, autogalaxy_workspace, autofit_workspace, autolens_workspace_test, HowToLens or euclid. It does appear in three autolens_profiling scripts (interferometer/{delaunay,rectangular}/likelihood_breakdown_numba.pyandmisc/likelihood_breakdown/interferometer_pixelized_numpy.py) and in notes; those scripts set it explicitly, so they keep their hard-gate meaning. The curated smoke passed, including the autolens interferometer scripts.Next: phase 2 is the autolens_profiling re-measurement record on both masks with the new default (
start_workspace), after this PR merges.Jammy2211 commentedon Oct 15, 2026
Shipped
Library phase merged in #632 (17caf22, 2026-10-15). Pending release.
interferometer_numba_nnz_per_source_max: 60gate is replaced by a two-constant cost model:t_fft = c_fft·S·pocketfft-work(padded 2ny×2nx),t_dc = c_dc·nnz·ny·nx(c_fft 7.62e-10, c_dc 1.77e-09; least squares on 9 paired timings = 6 autolens_profiling#226 bake-off cells + 3 Aris 512 points; right route at all 9, ratio residuals 0.68–1.47).interferometer_numba_nnz_per_source_maxdefaultnull= cost model; positive = hard gate; 0 = numba off. Newinterferometer_routesetting forces a route. Missing extent under the cost model → FFT.interferometer_numba_nnz_per_source_max: 60.0.Not shipped here: phase 2 (autolens_profiling re-measurement record), re-filed as Mind
draft/feature/autolens_profiling/interferometer_route_gate_phase2_remeasure.md. Campaign: PyAutoLabs/PyAutoPulse#62 (interferometer_large_mask_aris).Record: PyAutoMind
complete/2026/10/interferometer-numba-route-gate.md.