Repository navigation
Image QC metrics per plane and channel, computed in the histogram pass - #365
Merged
Merged
Conversation
Design metrics 1-8 (MorphoFeature image-qc-metrics-2026-11-21): Otsu foreground fraction (with threshold and separation), entropy, normalized variance, Tenengrad, Laplacian variance, correlation at lags 1/4/16, local focus median/p10 and tile mean CV. They are scale-free and measured from the blocks the histogram already reads (no extra read), and written to the new run-folder table image_qc.csv, which the manifest names under the optional key image_qc. The table follows the histogram step's reuse and resume rules (staging version 2). PLLS is left out because of its cost. Includes contract section 19, the user guide, the CHANGELOG and tools/benchmark_image_qc.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Resolves CHANGELOG (both entries kept) and manifest.py (main's docstring notes kept). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
B1 numpy>=2.0 floor (numpy.vecdot), with the other floors raised to releases that import with NumPy 2 (scipy 1.13, matplotlib 3.8.4, scikit-image 0.24 for he-mask). B2 a changed memory limit recomputes rows when planes are read in blocks (run.reuse _settings_changed) and staged rows are keyed by it (RowKey.block_limit). M1/M2/L5/L6 contract section 19, module docstring and user guide: Otsu separation baseline 2/pi and correlation_1 as the emptiness marker, scale-free only under multiplication, tenengrad = normalized_variance x (1 - rho_2), local_focus_p10 as content coverage, central-difference Tenengrad and Pearson correlations. L3 the QC sums' memory (qc_memory, band sized min(64, rows) + reach) is in the plane and block estimates (plane_binning.Binning). L4 float planes are centred on their mean. L7 manifest.py and run/pipeline.py split cleanly (manifest_files, run/state, run/outputs); the docstring notes the squeeze removed are back. F1 blank_reasons column. F2 block tolerances are observations, with a boundary test. F3 the reader and writer refuse inconsistent rows. F4 reuse compares the full row identity. F5 blocked-reader test with 16-row blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…ock; codec test limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…unt the QC working memory The run's output code moved from pipeline to run.outputs, so the clock and the histogram and manifest writers are patched there. The whole-plane estimate now includes qc_memory; the block-read tests take their limit from block_limit, the small plane's estimate with the QC term of one row of the large plane, so the large plane is still read in blocks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per-plane, per-channel image QC metrics (design metrics 1-8), measured in the histogram pass from the blocks the histogram already reads (no extra file read), written to a new run-folder table
image_qc.csvand named in the manifest (image_qc: file, SHA-256,qc_version, settings).image_qc.py(pure):PlaneQcaccumulates combinable sums block by block;finish(histogram)adds the histogram metrics;ImageQcholds the values (None = undefined, written blank, never zero).image_qc_sums.py(pure):BlockSums, one cache-blocked sweep per block. Every neighbour metric is reduced to shifted dot products (10 offsets) plus thin border strips, so the cost is one pass over memory.image_qc_moments.py(pure):Moments/CoMoments, combined across blocks exactly (pairwise Chan update).image_qc_table.py(io): theimage_qc.csvwriter and strict reader.image_qc_record.py(pure): the manifest entry, read strictly.run/rows.py:_measuretaps the histogram's first pass over each channel plane.planes.pyis not touched.<id>.qc.csvis staged with the rows (version 1 records are discarded and those images read again);qc_versionor settings, or not matching the histogram table recomputes every row, with the reason in the report.run/generation.py,run/pipeline.py,log_target.py: the table is part of the generation and protected from--log.Not touched:
channel_identity.py(PR #364) andnotebook_channels_repeat.py. Feeding the shape metrics into channel identity is a later PR.Design (binding): MorphoFeature
claude/restart-doc,docs/restart/designs/image-qc-metrics-2026-11-21.mdMetric definitions (
qc_version1)m= plane mean. Every ratio metric is divided bymorm², so it is scale-free. Ratio metrics are blank whenm <= 0.otsu_thresholdotsu_separationforeground_fractionentropynormalized_variancevar / m²(see choices)tenengrad(mean gx² + mean gy²) / m², central differences(I[x+1]-I[x-1])/2laplacian_variance/ m²correlation_1/4/16local_focus_median,local_focus_p10/ m²(plane mean) on a fixed 8 x 8 grid; median and 10th percentile (linear) over 64 tiles; blank when a side is < 16 pxtile_mean_cvqc_statusis one of:measured;non_finite_pixels: pixel metrics blank, histogram metrics on the finite pixels;no_finite_pixels: all blank.For blocked planes (
read_in_blockstrue), the histogram, moment and tile metrics equal the whole plane's. The neighbour metrics leave out pairs that cross a block edge (tested: within 5% for Tenengrad and the Laplacian, and within 0.05 absolute for the correlations, on blocks down to 17 x 33 px).Benchmark
tools/benchmark_image_qc.py(committed). Setup:mode="full", 1 worker;origin/main68a3952 throughPYTHONPATH;run.steps.new_rows(read, histogram, QC, staging).plane_histograms.csvandexperiment_design.csvare byte-identical in all 40 runs, and all runs with the metrics wrote the sameimage_qc.csv(identical_outputs: true). The machine was shared (load average 1.2-2.4 during the runs; not pinned).The median is slightly over the +30% target (+32% whole, +39% blocks; noisy machine). The owner decides whether that is acceptable (see choices).
What was tried, from earlier rounds on the same data:
numpy.vecdotover rows of 8192 (final kernel)The final kernel's own cost is about 19-26 ms per 2048² plane.
PLLS is not included. Measured alone,
scipy.fft.rfft2plus power on one 2048 x 2048 plane takes 29 ms (float32) to 62 ms (float64), before any radial averaging. That is +40-85% of the ~74 ms per channel plane histogram step on its own, so the total cannot stay within +50%. This is a micro-benchmark of the FFT only, not a pipeline run.Choices for the owner
normalized_variance=var / m², not CellProfiler'svar / m. The design names CellProfiler's FocusScore (var / m), but that scales with intensity, which contradicts the design's scale-free principle and its brightness-invariance validation. Alternative: keepvar / mand exclude it from the invariance rule.otsu_separation. Two-class Otsu always splits, so on a truly empty field (unimodal noise)foreground_fractionis about 0.5, not near 0 as the design's validation expects. A near-empty field (noise plus a speck of debris) does give < 1%.otsu_separation(Otsu's own separability) tells the two cases apart (testtest_otsu_on_noise_alone_splits_the_noise_with_a_low_separation). Alternative: drop it and document the limitation only.foreground_fractionwithin 0.01,entropywithin 0.05).non_finite_pixels), rather than a masked computation.image_qcentry (every existing output folder), the first rerun recomputes every row ("the earlier run has no image QC table"), rather than rereading only for QC.channel, sorted by image, z, t, channel ID. Population statistics are used (ddof 0); entropy is in bits.RunInputsdoes not read or checkimage_qc.csv, and stale-folder refusals ignore it, since the table is information only. If later bricks consume it,RunInputsshould check its SHA-256.Spec impact
None needed. The Spec checkout (MorphoBricks-Spec
main232fb38) does not enumerate the preprocessing run folder's files:run_manifest.jsonand the histogram table only.RUN_FOLDER_STALElists the filesRunInputschecks, which this PR does not extend.So there is no Spec PR. If the owner wants the table enumerated (for example when channel identity starts using it), a one-line addition to ADR 0007's validated-run-folder context would do.
Tests
tests/test_image_qc.py(27 tests):tests/test_image_qc_run.py(22 tests):read_in_blocks, identical histograms;--logprotection;Local check
python tools/check.pyin the worktree with a fresh venv (.[dev,notebook,zarr]), noPYTEST_ADDOPTS, freshTMPDIR. Ruff check and format pass, and the code rules pass (the baseline is lowered by one line fortests/test_notebook_ui.py). Pytest: 6840 passed, 16 skipped, 0 failed. The skips are environment skips:he-maskextra);Not verified:
https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Generated by Claude Code