Skip to content

Segmentation hand-off: read channels by ID and write masks as preprocessing expects - #367

Merged
guijacquemet merged 14 commits into
mainfrom
claude/segmentation-handoff
Nov 22, 2026
Merged

guijacquemet merged 14 commits into
mainfrom
claude/segmentation-handoff

Conversation

@guijacquemet

@guijacquemet guijacquemet commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Step 1 of the Cellpose notebook plan: a tested hand-off helper (numpy and tifffile only; no Cellpose import). A segmentation tool reads channels by canonical ID from a described preprocessing run and writes masks where, and as, preprocessing expects them.

Design: https://github.com/CellMigrationLab/MorphoFeature/blob/claude/restart-doc/docs/restart/designs/cellpose-notebook-2026-11-21.md

API (segmentation_handoff.py, segmentation_record.py, lock_file.py; exported in __init__)

  • images_to_segment(inputs, mask_type, masks_root): images without a mask of that type, decided by the run's own discovery and rules (masks/files.py: folder masks and OME-Zarr label images; _name_issues, _description_name_issues, _duplicate_issues), so it cannot disagree with the next run.
  • resolve_segmentation(inputs, channel_ids) -> SegmentationPlan (plan.read(fov)) and segmentation_input(...): channels by ID at each image's own C index, resolved once per batch; resolver refusals verbatim (a channel some images lack is A_SCOPE_INCOMPLETE); Z/T > 1 refused.
  • mask_path, write_mask(masks_root, mask_type, fov, labels, inputs, *, overwrite=False, on_published=None): 2D, image Y/X, integer or boolean, 0..2,147,483,647; zlib int32 YX TIFF; folder rules; twin check before the lock (a refusal creates nothing), then one exclusive per-type write lock (<masks>/<type>/.segmentation_write.lock, refused at once when held, never waited on) around a second twin check, the exclusive claim, the publish and the optional on_published callback, in both modes.
  • SegmentationRecord / SegmentationBatch / SegmentedImage, write_record (append-only history of batches; a batch ID may only grow; per-image mask SHA-256; record lock = the run's own lock), read_record, mask_sha256, new_batch_id() (UTC second + random suffix), check_mask_type (check_identity_text, NFC, one-folder rule).
  • lock_file.py: the run's exclusive lock core (exclusive_lock, LockHolder with a random token compared on removal, read_holder), shared by run/lock.py and both segmentation locks.

The masks step skips segmentation_record.json only directly in a type folder; lock and temporary files are dot-files, skipped by discovery. Contract §15.5 documents the limits: the locks protect writers on one computer, not two Colab sessions on one Drive folder; a reader may briefly see a zero-byte placeholder; do not run preprocessing on the masks folder during a batch.

Reviews

Tests

tests/test_segmentation_handoff.py, tests/test_segmentation_write_lock.py, tests/test_segmentation_record.py (channels by ID on a mixed 3/4-channel run, round trip through run_experiment(..., masks_folder=...) with no issues, twins in either layout/case/OME-Zarr, name parity, held locks in both modes, the mixed-writer interleaving, callback under the lock, record history, lock tokens). Each audit fix has a test that failed on the previous head.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm

…essing expects

Add segmentation_handoff (images_to_segment, segmentation_input, mask_path,
write_mask) and segmentation_record (versioned provenance file). The masks step
skips segmentation_record.json silently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…ly, resolve once

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…te checks, no silent replace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…d links needed

Without overwrite the target name is claimed with O_CREAT|O_EXCL and the finished file is
moved over our own empty placeholder, so two writers never replace each other's mask on any
file system (a Google Drive mount has no hard links). A failed move removes the placeholder
only while it is still empty. Contract 15.5 now matches the code (it still described a
hard-link fallback).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…ase-insensitive file system passes

On Windows and macOS the twin spelled in another case is the mask path itself, so
'the mask path does not exist' cannot hold there. Assert instead that the only
file under the masks folder is the untouched twin (stricter on every platform).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…cked record writes, unique batch IDs

The pairing now also counts the label images inside OME-Zarr images and applies the run's own
_name_issues and _duplicate_issues instead of a second copy of those rules, so an OME-Zarr image
with a labels/<type> mask is not listed and write_mask refuses it. write_record holds an
exclusive lock file while it rewrites the record. new_batch_id() makes a unique batch ID.
The contract says the exclusive create protects writers on one computer only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… never remove a lock another writer holds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…lock is removed only by its writer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…, one write lock per type in both modes, a random token in lock holders

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…er cleanup after the lock

A refusal before the lock creates nothing; the check under the lock catches a mask
published meanwhile. The cleanup that ran after the lock was released could remove a
folder another writer had just created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
@guijacquemet
guijacquemet marked this pull request as ready for review November 22, 2026 18:05
@guijacquemet
guijacquemet merged commit 7cfe8e4 into main Nov 22, 2026
8 checks passed
@guijacquemet
guijacquemet deleted the claude/segmentation-handoff branch November 22, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants