Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
cdb592d
Cellpose glue for the segmentation hand-off (step 2): verified model,…
claude Nov 22, 2026
6a9765d
Cellpose glue: batch IDs to the microsecond, refusals shown verbatim,…
claude Nov 22, 2026
96acdc0
Cellpose notebook on a described run (step 2 of the segmentation hand…
claude Nov 22, 2026
ad3500c
Merge remote-tracking branch 'origin/claude/segmentation-handoff' int…
claude Nov 22, 2026
be8342f
Cellpose notebook: verify every weight's device, record the full run,…
claude Nov 22, 2026
52996b8
Pin the Cellpose notebook to the commit that holds its code (be8342f)
claude Nov 22, 2026
4864361
Notebook tests: set form fields with a function, so Windows paths kee…
claude Nov 22, 2026
0a26163
Merge remote-tracking branch 'origin/claude/segmentation-handoff' int…
claude Nov 22, 2026
b70b138
Pin the Cellpose notebook to 0a26163253dd, which holds its code after…
claude Nov 22, 2026
d66898c
Merge remote-tracking branch 'origin/claude/segmentation-handoff' int…
claude Nov 22, 2026
7045ece
Cellpose batch: record each mask before the next image, withdraw an u…
claude Nov 22, 2026
a3789b2
Pin the Cellpose notebook to 7045eceef439, which holds its code after…
claude Nov 22, 2026
72f4b55
Merge remote-tracking branch 'origin/claude/segmentation-handoff' int…
claude Nov 22, 2026
a2807f6
Cellpose batch: withdraw on interrupt, stamp and hash under the write…
claude Nov 22, 2026
df63281
Cellpose notebook: re-pin RELEASE_REF to the commit holding the final…
claude Nov 22, 2026
ffdca16
write_mask: roll back when the caller's provenance step fails
claude Nov 22, 2026
e151a4c
Merge remote-tracking branch 'origin/claude/handoff-rollback' into cl…
claude Nov 22, 2026
dcb6b83
Cellpose batch: hash and record each mask under the type's write lock…
claude Nov 22, 2026
65be299
Cellpose notebook: re-pin RELEASE_REF to the commit holding the final…
claude Nov 22, 2026
a585f59
write_mask: decide the rollback from the file system; report stray ba…
claude Nov 22, 2026
42b1f4d
Merge remote-tracking branch 'origin/claude/handoff-rollback' into cl…
claude Nov 22, 2026
425ca3e
Cellpose batch: keep the mask when its record was saved; re-raise int…
claude Nov 22, 2026
fd87c7e
Cellpose notebook: re-pin RELEASE_REF to the commit holding the final…
claude Nov 22, 2026
1b7f4e5
Docs: write_mask rolls back on every callback error; record is not fs…
claude Nov 22, 2026
eda2bdc
Merge remote-tracking branch 'origin/claude/handoff-rollback' into cl…
claude Nov 22, 2026
c1977aa
Cellpose batch: recover a failed record write only on the full batch …
claude Nov 22, 2026
ee388ff
Cellpose notebook: re-pin RELEASE_REF to the commit holding the final…
claude Nov 22, 2026
cc649de
Cellpose recovery: require the batch's whole image list (a shortened …
claude Nov 22, 2026
ae3347e
Pin the Cellpose notebook to cc649def101e, which holds its code
claude Nov 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,15 @@ src/morphobricks_preprocessing/
notebook; numpy and tifffile only (contract §15.5); segmentation_record.py
SegmentationRecord/SegmentationBatch and the append-only history file
<masks>/<type>/segmentation_record.json (mask type checked like a discovered one)
segmentation_cellpose.py the Cellpose glue (step 2; cellpose, torch and dinov3 imported only inside
functions, Cellpose 4.2.1.1 pinned, not a dependency): Settings, diameter_from_text,
eval_arguments (channel_axis=0), segment_image, run_batch (every image written, skipped
or failed in a SegmentationBatch), channel_coverage (the preview reuses preview_render's display_panel and outline);
segmentation_cellpose_model.py choose_model (a built-in name or an existing file,
before loading), load_model (the weights file and device verified after loading, no
download without allow_download, SHA-256 recorded, DINOv3 at a pinned commit);
notebook_cellpose.py CellposeWorkflow, ChannelPicker, CellStop: the widgets behind
notebooks/MorphoBricks_Cellpose.ipynb
mask_conversion.py RunInputs.binary_to_instances / classes_to_binary (Spec ADR 0005 BM12, step
2b): new mask files from a checked mask, refused before any read or write
(check_class_image, collisions, idempotent retry); mask_lineage.py their
Expand Down
83 changes: 83 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ v0.5.1: no run before v0.5.1 is read.)

## Unreleased

- Cellpose batch recovery after a failed record write also requires the batch's whole image list to be the one written (a reused batch ID that leaves out earlier images is refused and the mask restored; ChatGPT al FILE 1).

**One header-name rule (#368).** Whether a raw header name is a confirmed name of a channel is now
decided by one function, `description_identity.header_name_matches` (equal after NFC, never trimmed,
never case-folded), used by the notebook's channel matrix, the fast layout, the physical check and
Expand All @@ -23,6 +25,61 @@ old pin are refused with `spec_commit_mismatch`, as at every re-pin.

### Added

- **Cellpose notebook, a reused batch ID is not recovered as the old batch's (external audit B1,
22 November 2026).** When the record write failed, the recovery check compared only the latest
entry's batch ID and mask SHA-256, so a reused ID with another model or settings and identical
mask bytes (refused by `write_record`) was reported `written` under the earlier batch's
provenance. The check now needs the record's batch of that ID to have the expected batch's
provenance (`SegmentationBatch.same_provenance`) and the image's latest entry to be exactly the
expected one; otherwise the error propagates and `write_mask` rolls back. The earlier-mask note
still comes from a `stray_backups` diff read under the lock (`on_published` is not told whether
its writer made a backup; changing its signature is outside this fix).
- **Cellpose notebook, provenance under the write lock (external audit of PR #375, 22 November
2026).** Each mask's SHA-256 and its record append now run in `write_mask`'s `on_published`,
under the type's write lock (then the record's lock; never the reverse), so two overlapping
batches can no longer leave a latest record entry naming one batch while the bytes on disk are
the other's. A hash, newest-entry or record failure (an interrupt too) makes `write_mask` roll
back under that lock (#397): an overwritten mask is restored byte for byte, a new one removed.
`_withdraw`, its identity stamp and its own (wrong-folder) lock are gone. An existing mask not
shown to be its latest recorded one is `needs_attention` in the table and counts, not
`skipped`. The fail-safe tests compare on-disk SHA-256 with the latest record entry.
- **Cellpose notebook, review fixes (PR #375)**: the per-image hash/record step now catches
`BaseException`, so a `KeyboardInterrupt` (stop button) between `write_mask` and the image's record
write withdraws the mask (identity-checked) and then re-raises; the identity stamp and SHA-256 are
taken under `write_mask`'s write lock (`on_published`, PR #367); a retry compares a recorded mask's
SHA-256 with its latest record entry and reports "exists, but differs from its recorded provenance".
- **Cellpose notebook, failure-safe record (external audit C6).** A published mask is never left
without its record entry: each mask is hashed and recorded right after publication, before the
next image; if that fails the mask is removed (only if still the file written; otherwise a
`WITHOUT provenance` warning names it), the image is `failed`, the remaining images are
`failed` as not started and the batch stops. A retry reports a mask without a record entry
as `exists without provenance` instead of a plain skip. A diameter giving fewer than one flow
iteration is refused instead of clamped to 1; the docs and notebook say the verified device is
the network's placement, not Cellpose's CPU post-processing. `Settings` moved to
`segmentation_cellpose_settings` (re-exported).
- **Cellpose notebook** (`notebooks/MorphoBricks_Cellpose.ipynb`, step 2 of the segmentation
hand-off; owner request of 21 November 2026): opens a described run, chooses the mask type and
the channels **by channel ID** (dropdowns; a channel only some images have is refused with
the resolver's `A_SCOPE_INCOMPLETE` message, shown unchanged), loads **Cellpose 4.2.1.1**
(`cpsam_v2` by default, `cpsam`, `cpdino`, `cpdino-vitb`, or a model file), previews one image,
segments every image without a mask and ends with a table of every image as written, skipped
or failed with its error, then says how to declare the new mask type and rerun preprocessing.
Cellpose is not a dependency: `segmentation_cellpose`, `segmentation_cellpose_model` and
`notebook_cellpose` import `cellpose`, `torch` and `dinov3` only inside functions. What
Cellpose 4.2.1.1 does silently is refused: a model name or file it cannot find becomes
`cpsam_v2` (the model is validated before loading and the loaded weights file must be the one
asked for), a missing GPU becomes the CPU (the device is stated, an unavailable one is refused,
and the model must sit on it), uncached weights are downloaded (only with the download box
ticked, after the notebook says they come from Hugging Face), channels beyond the first three
are dropped by position (more than three are refused), and no diameter means the model's native
scale, not an estimate (a blank diameter is recorded as the native scale, 0 is refused). The
SHA-256 and file name of the weights, every effective setting and the DINOv3 commit
(`cpdino` models, installed only when ticked, `--no-deps`, at a pinned commit that is not
verified against `cpdino`) go into the `SegmentationBatch`, with the full normalisation, the
flow iterations, whether the weights were downloaded and from where, and a note that Cellpose may
run its flow-threshold step on the CPU (and the masks on the CPU on `mps`). The model's device is
verified on every parameter and buffer of the network. The batch ID is `new_batch_id()`. 2D
images only.
- **Segmentation hand-off** (owner request of 21 November 2026, step 1 of the Cellpose notebook
plan): `segmentation_handoff` and `segmentation_record` (numpy and tifffile only; no
segmentation tool is imported). `images_to_segment` lists the run's images that have no mask
Expand Down Expand Up @@ -50,6 +107,32 @@ old pin are refused with `spec_commit_mismatch`, as at every re-pin.
from an older run, still reads, token `""`). `write_mask(..., on_published=callable)` calls it with the
published path while the write lock is still held (not when refused; its error propagates, the mask
stays), so a caller can hash or stamp the file before another writer can replace it.
- **`write_mask` rolls back when `on_published` fails** (external audit of #375, 22 November 2026).
The caller's provenance step now runs in the callback under the type's write lock; if it raises
(any exception, `KeyboardInterrupt` too), `write_mask` restores an overwritten mask byte for byte
from a hidden copy kept beside it (`.<name>.previous.<random>`, skipped by the masks step, removed
on success) or removes a new mask, still under the lock, then re-raises. A rollback that cannot
restore or remove raises `SegmentationError` naming both files and keeps the copy. Before, a
failed overwrite left the new mask published and the earlier valid mask was lost. The record's
duplicate-image check is linear (a seen-set instead of `list.count` per entry).
- **Docs: `on_published` rollback and record durability** (external audit, 22 November 2026): the
`write_mask` docstring and contract §15.5 say that the rollback happens on every callback exception
even after the callback committed (it must reconcile its own post-commit failures), and that the
record is replaced atomically but not fsynced (not a power-loss guarantee). Behaviour unchanged.
- **`write_mask` decides its rollback from the file system** (review of #397): an interrupt right
after the move but before `write_mask` returned used to delete the copy of the earlier mask, so the
new mask stayed and the earlier one was lost; now the target's inode says whether the move happened
and the earlier mask is restored (a new mask removed). New `stray_backups(masks_root, mask_type)`
lists `.<name>.previous.<random>` copies left by killed processes; they are reported, never deleted.
- **Cellpose batch: the mask and its record stay together when `write_record` fails after saving**
(review of #375). If the record file was saved and then the call raised (`OSError` releasing the
lock, `KeyboardInterrupt` after the replace, or an interrupt before the entry was noted), the
callback re-reads the record; when the image's latest entry is this batch with this SHA-256 the mask
is kept (not rolled back), an error stops the batch and an interrupt is raised again with the image
in the final record. An interrupt after a successful commit is no longer swallowed. Whether an
earlier mask was replaced (the table's "restored" or "removed" note) is read under the write lock.
The notebook's batch summary lists stray `.previous` copies (`stray_backups`), never deleting them.
New `segmentation_cellpose_commit` module (`check_newest` moved there).
- **Channel-identity check after the run** (contract §14.1, design
`channel-identity-check-2026-11-21` in MorphoFeature's `claude/restart-doc`): from the image
summaries the intensity checks already make (no pixel read), every imaging unit's channel is
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@ Pre-alpha; the current release is v0.8.0 (what is planned next is in
- Running an experiment end to end, updating earlier output incrementally, resuming
interrupted runs and publishing results in one crash-consistent step
- Reading a run from another brick: `RunInputs.open(output_folder)`
- A Cellpose notebook ([notebooks/MorphoBricks_Cellpose.ipynb](notebooks/MorphoBricks_Cellpose.ipynb))
that makes masks for a described run, channels chosen by channel ID (Cellpose 4.2.1.1; not a
dependency of the brick)
- A step-by-step notebook ([notebooks/MorphoBricks_Preprocessing.ipynb](notebooks/MorphoBricks_Preprocessing.ipynb))
and a command line (`morphobricks-preprocessing check` / `run` / `masks` / `calibrate` /
`describe` / `convert`)
Expand Down
46 changes: 45 additions & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -1352,14 +1352,58 @@ links (a Google Drive mount) too.
the image to the published file, **in both modes**: an overwriting writer and a default writer
can never replace each other's mask. A held lock is refused at once, naming its holder, and
never waited on; a killed writer leaves it, and the message says to delete it. Like the
record's lock it is a dot-file, skipped by the masks step. The optional `on_published(path)` callback runs right after the publish, still under the lock (take a hash or file stamp there; it is not called when the write is refused, and an error it raises propagates with the mask left published). A lock holder carries a random
record's lock it is a dot-file, skipped by the masks step. The optional `on_published(path)` callback runs right after the publish, still under the lock (take a hash and write the provenance there; it is not called when the write is refused). If it raises (any exception, `KeyboardInterrupt` too), the write is **rolled back under the same lock** and the error propagates: with `overwrite=True` the earlier mask was first copied to a hidden file beside it (`.<name>.previous.<random>`, a dot-file the masks step skips, removed once the callback returns) and is moved back over the new one byte for byte, with its mode; a new mask is removed. A rollback that cannot restore or remove is a `SegmentationError` naming both files, and the copy is kept. The rollback is unconditional: it happens on **every** exception from the callback, even one raised after the callback had durably committed its provenance, so a callback that commits must reconcile its own post-commit failures (keep the mask and not raise, or undo its commit) before it raises; `write_mask` has no committed-state protocol. The provenance record is written atomically (a temporary file replaced into place) but **not fsynced**: that protects readers from a half-written record, not a power loss, after which a record or mask may be missing or stale. A lock holder carries a random
token, so a lock deleted and retaken is never removed by its earlier holder. Limits: the lock
protects the writers of one computer (not a Drive folder open in two Colab sessions); a
reader may briefly see a **zero-byte placeholder** at the mask's name while a mask is being
written (the claim before the move); and preprocessing must **not be run on the masks folder
during a batch** (a run could read that placeholder or a half-written batch). A placeholder
left by a killed writer is deleted by hand.

The shipped Cellpose notebook builds on this hand-off and adds no rule of its own to the masks
(`segmentation_cellpose`, `segmentation_cellpose_model`; Cellpose, `torch` and `dinov3` are
imported only inside functions, so importing the brick and discovery stay inert). Cellpose
4.2.1.1 is pinned and is not a dependency. Each of its silent behaviours is a refusal here: a
model is one of `cpsam_v2`, `cpsam`, `cpdino`, `cpdino-vitb` or an existing file, checked before
loading, and the weights file the loaded model reports must be the one asked for; a device
(`cuda`, `mps`, `cpu`) is always stated and an unavailable one is refused. After loading, **every parameter and every
buffer of the loaded network (`model.net`) must be on a device of the requested type** (`cuda:1`
counts as `cuda`; the index is not compared); a network that lists no tensors is refused.
Cellpose's own `model.device` is not evidence (it only repeats the device it was given), and
nothing beyond the network's tensors is verified. The device is the placement of the network's tensors; Cellpose's own post-processing is not
covered by it: Cellpose may move the flow-threshold quality step to the CPU when GPU memory is
short, and always computes the masks from the flows on the CPU on `mps`; that is Cellpose's
behaviour, **not refused**, and the batch records it as a note. Uncached weights are downloaded only on an explicit request; more than
three channels are refused (Cellpose reads the first three by position); a blank diameter is
the model's native scale, never an estimate and never 0, and a diameter that gives fewer than one
flow iteration (`int(200 / rescale)` is 0) is refused, never raised to 1. The batch's `model` holds the model
name, the weights file name (or the custom file's path) and its SHA-256, and its `parameters`
every effective setting (diameter in pixels or null with its meaning, thresholds, the **full**
normalisation dictionary given to Cellpose (its defaults read from its module at call time, with
the user's values applied) and the flow iterations `niter` (`int(200 / rescale)`, passed
explicitly), batch and tile size, `channel_axis` 0, device type, versions, `weights_downloaded`
and `weights_source` (the Hugging Face URL Cellpose uses; null for a model file), the device
`notes` and, for a DINOv3 model, the DINOv3 commit). The batch ID is the record's
`new_batch_id()`. The DINOv3 package is installed with `--no-deps` at a pinned commit, the head
of its default branch on 22 November 2026, which is **not verified against `cpdino`**; Cellpose
imports it only when first imported, so the runtime is restarted after installing it.
Two-dimensional images only. **A mask the batch publishes is never left without its record
entry**: each mask's SHA-256 and the record write (the batch so far, the same batch ID growing)
happen in `write_mask`'s `on_published`, **under the type's write lock**, before the next image
starts (lock order: the type's write lock, then the record's lock; never the reverse). So a
second batch cannot publish this type between a mask and its record entry, and the latest record
entry of an image always names the bytes on disk. An entry that would not be the image's newest
(a batch recorded after this one started already wrote the image) is refused. If hashing,
that check or the record write fails, even by an interrupt, `write_mask` rolls the write back under
the same lock (an overwritten mask restored byte for byte, a new one removed; a rollback that
cannot is a loud error naming the files), the image is `failed` with the error and `rolled back:
earlier mask restored` or `rolled back: the new mask was removed`, the images not yet started are
`failed` as `not started`, and the batch stops (`BatchResult.stopped`). A retry without
`overwrite` shows an existing mask as `skipped` only when its SHA-256 is that of its latest
`written` entry; otherwise (no entry: `exists without provenance (not made by a recorded batch)`,
another SHA-256, a twin elsewhere, or not readable) it is `needs_attention` in the table and the
counts (recorded as `skipped` with that reason); the model that made it is not guessed.

## 16. Shared rules for other bricks (MorphoBricks D86)

Rules every MorphoBricks brick applies in one way, so they live here as public functions
Expand Down
Loading
Loading