Repository navigation
Cellpose notebook on a described run (step 2 of the segmentation hand-off) - #375
Draft
guijacquemet wants to merge 29 commits into
Draft
guijacquemet wants to merge 29 commits into
guijacquemet wants to merge 29 commits into
Conversation
… device and weights segmentation_cellpose and segmentation_cellpose_model import Cellpose, torch and DINOv3 only inside functions; Cellpose 4.2.1.1 is pinned and is not a dependency. A model is one of the four built-in names or an existing file, validated before loading; after loading the weights file and the device must be the ones asked for (Cellpose would substitute cpsam_v2, fall back to the CPU or download weights silently); the weights are hashed for the record. Channels come from resolve_segmentation as a (C, Y, X) stack with channel_axis=0 (at most three); a blank diameter is recorded as the model's native scale, never 0. run_batch writes each mask with write_mask and records every image as written, skipped or failed in the existing SegmentationBatch. notebook_cellpose holds the widgets behind the notebook. Tests run against a stand-in for Cellpose and torch (tests/cellpose_stub.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… fixture shared A batch started twice within one second got the same batch_id and was refused by the record; the ID now includes microseconds. The channel dropdowns show the resolver's multi-line refusal unchanged. The mixed-channel run is a shared fixture (mixed_run) for the Cellpose tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…-off) notebooks/MorphoBricks_Cellpose.ipynb: install (Cellpose 4.2.1.1 and the preprocessing commit with this code, pinned), Google Drive, open a run and a masks folder, mask type and channels by ID (dropdowns; a partial channel is refused with the resolver's message, shown unchanged), model and device (explicit), diameter and thresholds, preview with an overlay, batch with progress and a table of every image, and the next steps. Every code cell is one call on CellposeWorkflow. Headless tests run the cells on a synthetic mixed-channel run with a stand-in model, including the rerun of preprocessing on the masks. Docs: user guide, contract 15.5, README, AGENTS.md map, CHANGELOG, notebook version file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…o claude/cellpose-notebook
… reuse display code - Device: every parameter and buffer of the network must sit on the requested device type (model.device only repeats what Cellpose was given); the stand-in has a network now. - Record: weights downloaded in this session and their source, the full normalisation (Cellpose's defaults read at call time, copied) and the flow iterations (passed explicitly), and a note that Cellpose may run its post-processing on the CPU (documented, not refused). - The channel choice is resolved again before a preview or a batch (one refusal, nothing starts); an error while writing the record never hides the error that interrupted the batch. - Batch IDs come from new_batch_id(); the preview reuses preview_render's display_panel and outline. - Colab: restart notes for numpy/opencv and for DINOv3 (installed with --no-deps; the pin is the 22 Nov 2026 branch head, not verified against cpdino). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…p their backslashes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…o claude/cellpose-notebook
…merge of the base Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…o claude/cellpose-notebook
…nrecordable one A mask was published before it was hashed and the record was written only after all masks, so a failed hash left a "failed" image with a mask, and a failed final write left every mask without provenance (a retry then skipped them). Now each mask is hashed and recorded right after it is published; on a failure it is removed (only while still the file written), the image is failed, the rest are failed as not started and the batch stops. A retry reports a mask with no record entry as "exists without provenance". A diameter that gives fewer than one flow iteration is refused, not clamped to 1; the device wording separates the verified network placement from Cellpose's CPU post-processing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…failure-safe record Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…o claude/cellpose-notebook
… lock, retry checks the recorded hash Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… src Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Under the type's write lock, an overwrite first copies the earlier mask to a hidden .<name>.previous.<random> beside it. If on_published raises (any BaseException), the earlier mask is moved back byte for byte, or a new mask is removed, still under the lock, then the error propagates. A rollback that cannot restore or remove raises SegmentationError naming both files and keeps the copy. The record's duplicate-image check uses a seen-set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…aude/cellpose-notebook
…; roll back via write_mask The SHA-256 and the record append now run in write_mask's on_published, under the type's write lock (then the record lock), so overlapping batches cannot leave a latest entry naming one batch while the bytes are another's. An entry that would not be the image's newest is refused. A failure there (an interrupt too) makes write_mask roll back: an overwritten mask is restored, a new one removed. _withdraw and its wrong-folder lock are gone. An existing mask not matching its latest entry is needs_attention in the table, not skipped. Fail-safe tests compare on-disk SHA-256 with the latest record entry; new orchestrated A/B interleaving tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… src Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…ckups An interrupt after os.replace but before _write_and_publish returned dropped the backup while the new mask stayed. The target's inode now decides whether the publish happened. stray_backups() reports leftover .previous copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…aude/cellpose-notebook
…errupts; report stray backups If write_record saved the file and then raised, the callback re-reads the record and, when the image's latest entry is this batch with this SHA-256, keeps the mask instead of rolling back. An interrupt after a successful commit is raised again. The replaced/removed note is read under the write lock. The batch summary lists stray backup copies. check_newest moves to segmentation_cellpose_commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… src Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…ynced The write_mask docstring and contract 15.5 now say that the rollback is unconditional (a callback that commits must reconcile its own post-commit failures) and that the record is replaced atomically but not fsynced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…aude/cellpose-notebook
…provenance The recovery check compared the batch ID and mask SHA-256 alone, so a reused ID with another model or settings (refused by write_record) was reported written under the earlier batch's provenance. It now needs the expected batch's provenance (same_provenance) and the image's exact latest entry; otherwise write_mask rolls back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
… src Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
…reuse of the ID is refused) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Step 2 of the segmentation hand-off (step 1 is #367, this PR is based on its branch):
notebooks/MorphoBricks_Cellpose.ipynbopens a described run, chooses the mask type and one to three channels by channel ID, loads Cellpose 4.2.1.1 (cpsam_v2default,cpsam,cpdino,cpdino-vitb, or a model file), previews one image with an overlay, segments every image without a mask and ends with a table of every image (written / skipped / failed with the error), then tells the user how to declare the mask type and rerun preprocessing.Cellpose is not a dependency:
segmentation_cellpose,segmentation_cellpose_modelandnotebook_cellposeimportcellpose,torchanddinov3only inside functions (a test checks that importing them imports none of the three). 2D only: Z or T above 1 fails per image with the hand-off's message.New files:
segmentation_cellpose.py(settings,segment_image,run_batch),segmentation_cellpose_model.py(model validation and verified loading; split to keep files under 600 lines),notebook_cellpose.py(widgets andCellposeWorkflow, so every notebook cell is one call), the notebook,tests/cellpose_stub.pyand tests.Reused from the hand-off (nothing re-implemented)
resolve_segmentation/SegmentationPlan.read(channels by ID,A_SCOPE_INCOMPLETEpasses through),images_to_segment,write_mask,mask_path,check_mask_type,SegmentationBatch/SegmentedImage/write_record,new_batch_id()(batch IDs),mask_sha256,SegmentationError,_io.file_sha256,RunInputs.open,MANIFEST_FILENAME, the repo'snotebook_style/notebook_progresscards, andpreview_render'sdisplay_panel/outline/as_colourfor the preview (no copy of the contrast or outline code).Audit items
choose_modelaccepts only the four names or an existing file, before anything loads; a built-in model on disk is loaded by its absolute path; after loading,pretrained_modelmust resolve to the requested file, else refused.model.netmust be on the requested device type (cuda:1counts ascuda), else refused; a network that lists no tensors is refused. Cellpose's ownmodel.deviceonly repeats the device it was given, so it is not used as evidence, and nothing beyond the network's tensors is claimed. Cellpose may still move the flow-threshold quality step to the CPU when GPU memory is short and always computes the masks from the flows on the CPU onmps; that is Cellpose's behaviour, not refused, documented (notebook, docstring, contract §15.5) and recorded as a note.allow_download. The record holds the SHA-256, the model and file name, whether the weights were downloaded in this session and where from (weights_downloaded,weights_source: the Hugging Face URL Cellpose uses).cpdinomodels need DINOv3 at the pinned commit (below).Also:
channel_axis=0on the(C, Y, X)stack in requested-ID order; more than three channels refused; blank diameter means the native scale, 0 refused; every effective setting is recorded and equals whatevalwas called with, including the full normalisation dictionary (Cellpose's defaults read from its module at call time and copied, then the user's values) and the flow iterationsniter(int(200 / rescale), passed explicitly). The channel choice is resolved again before a preview or a batch, so a choice changed after loading gives one refusal and starts nothing; an error while writing the record never hides the error that interrupted the batch.Colab
Cell 1 stops and says to restart the runtime if pip upgraded numpy or opencv that the kernel already has loaded. Cell 5 says that after installing DINOv3 the runtime must be restarted before loading a
cpdinomodel (Cellpose imports DINOv3 only when it is first imported). DINOv3 is installed with--no-deps(its requirements are unpinned and could replace torch; Cellpose reaches it throughdinov3.hub.backbones, which imports only numpy and torch). The pin6876159a...is the head of the DINOv3 default branch on 22 November 2026 and is not verified againstcpdino; the notebook and docs say so.Tests
tests/test_segmentation_cellpose.py,tests/test_segmentation_cellpose_record.pyandtests/test_notebook_cellpose.py(stand-in Cellpose and torch insys.modules; the stand-in has a network with parameters and buffers). Withtest_notebook_file.py,test_segmentation_handoff.pyandtest_segmentation_record.py: 192 passed.tools/check.py --rules,ruff checkandruff format --checkpass. The fulltools/check.pywas not run here.External audit, C6 (failure-safe record)
Merged
claude/segmentation-handoff(050e98c). A published mask is never left without a record entry: each mask is hashed and the record (same batch ID, growing) is written right after publication, before the next image. If hashing or that write fails, the mask is removed only while it is still the file written (inode, mtime, size, under the type's write lock); otherwise aWITHOUT provenancewarning names it. The image isfailed, the remaining images arefailedas not started, and the batch stops (BatchResult.stopped, shown in the notebook). A retry reports a mask with no record entry asexists without provenance. A diameter giving fewer than one flow iteration is refused instead of clamped; device wording now separates the verified network placement from Cellpose's CPU post-processing.tests/test_segmentation_cellpose_failsafe.py(7 tests) fails on b70b138 and passes now. Known limit: withoverwrite, a withdrawn mask cannot restore the earlier one it replaced (the entry says so).Not verified
No real Cellpose, torch, GPU or Colab run. The
CellposeModelsignature,pretrained_model,device,net,backbone,MODEL_DIR,normalize_default, theniterderivation and the CPU post-processing behaviour were read from the 4.2.1.1 wheel source; the stand-in copies them.RELEASE_REFin the install cell is the commit holding the code the cells call (a test fails ifsrc/changes after it): move it when this branch changes or is squash-merged. The notebook still needs a smoke test in a fresh Colab runtime and in JupyterLab (docs/colab-release-check.md).https://claude.ai/code/session_01Scma6W3LC11s3qjXHZuGDm
Generated by Claude Code