Repository navigation
Segmentation hand-off: read channels by ID and write masks as preprocessing expects - #367
Merged
Merged
Conversation
…essing expects Add segmentation_handoff (images_to_segment, segmentation_input, mask_path, write_mask) and segmentation_record (versioned provenance file). The masks step skips segmentation_record.json silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…ly, resolve once Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…te checks, no silent replace Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…d links needed Without overwrite the target name is claimed with O_CREAT|O_EXCL and the finished file is moved over our own empty placeholder, so two writers never replace each other's mask on any file system (a Google Drive mount has no hard links). A failed move removes the placeholder only while it is still empty. Contract 15.5 now matches the code (it still described a hard-link fallback). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…andoff # Conflicts: # CHANGELOG.md
…ase-insensitive file system passes On Windows and macOS the twin spelled in another case is the mask path itself, so 'the mask path does not exist' cannot hold there. Assert instead that the only file under the masks folder is the untouched twin (stricter on every platform). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…cked record writes, unique batch IDs The pairing now also counts the label images inside OME-Zarr images and applies the run's own _name_issues and _duplicate_issues instead of a second copy of those rules, so an OME-Zarr image with a labels/<type> mask is not listed and write_mask refuses it. write_record holds an exclusive lock file while it rewrites the record. new_batch_id() makes a unique batch ID. The contract says the exclusive create protects writers on one computer only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… never remove a lock another writer holds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…lock is removed only by its writer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…, one write lock per type in both modes, a random token in lock holders Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…er cleanup after the lock A refusal before the lock creates nothing; the check under the lock catches a mask published meanwhile. The cleanup that ran after the lock was released could remove a folder another writer had just created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 1 of the Cellpose notebook plan: a tested hand-off helper (numpy and tifffile only; no Cellpose import). A segmentation tool reads channels by canonical ID from a described preprocessing run and writes masks where, and as, preprocessing expects them.
Design: https://github.com/CellMigrationLab/MorphoFeature/blob/claude/restart-doc/docs/restart/designs/cellpose-notebook-2026-11-21.md
API (
segmentation_handoff.py,segmentation_record.py,lock_file.py; exported in__init__)images_to_segment(inputs, mask_type, masks_root): images without a mask of that type, decided by the run's own discovery and rules (masks/files.py: folder masks and OME-Zarr label images;_name_issues,_description_name_issues,_duplicate_issues), so it cannot disagree with the next run.resolve_segmentation(inputs, channel_ids) -> SegmentationPlan(plan.read(fov)) andsegmentation_input(...): channels by ID at each image's own C index, resolved once per batch; resolver refusals verbatim (a channel some images lack isA_SCOPE_INCOMPLETE); Z/T > 1 refused.mask_path,write_mask(masks_root, mask_type, fov, labels, inputs, *, overwrite=False, on_published=None): 2D, image Y/X, integer or boolean, 0..2,147,483,647; zlib int32YXTIFF; folder rules; twin check before the lock (a refusal creates nothing), then one exclusive per-type write lock (<masks>/<type>/.segmentation_write.lock, refused at once when held, never waited on) around a second twin check, the exclusive claim, the publish and the optionalon_publishedcallback, in both modes.SegmentationRecord/SegmentationBatch/SegmentedImage,write_record(append-only history of batches; a batch ID may only grow; per-image mask SHA-256; record lock = the run's own lock),read_record,mask_sha256,new_batch_id()(UTC second + random suffix),check_mask_type(check_identity_text, NFC, one-folder rule).lock_file.py: the run's exclusive lock core (exclusive_lock,LockHolderwith a random token compared on removal,read_holder), shared byrun/lock.pyand both segmentation locks.The masks step skips
segmentation_record.jsononly directly in a type folder; lock and temporary files are dot-files, skipped by discovery. Contract §15.5 documents the limits: the locks protect writers on one computer, not two Colab sessions on one Drive folder; a reader may briefly see a zero-byte placeholder; do not run preprocessing on the masks folder during a batch.Reviews
ac(DO NOT MERGE, B5/B6) -> fixed;ah(DO NOT MERGE: name parity, mixed overwrite race) -> fixed;aire-check at 7a0a521: MERGE (answers in MorphoFeatureclaude/restart-doc,docs/restart/chatgpt-answers/).write_maskrolls back when the caller's provenance step fails, restoring an overwritten mask), used by the Cellpose notebook Cellpose notebook on a described run (step 2 of the segmentation hand-off) #375.Tests
tests/test_segmentation_handoff.py,tests/test_segmentation_write_lock.py,tests/test_segmentation_record.py(channels by ID on a mixed 3/4-channel run, round trip throughrun_experiment(..., masks_folder=...)with no issues, twins in either layout/case/OME-Zarr, name parity, held locks in both modes, the mixed-writer interleaving, callback under the lock, record history, lock tokens). Each audit fix has a test that failed on the previous head.🤖 Generated with Claude Code
https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm