Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,13 @@ src/morphobricks_preprocessing/
masks.binary.label_binary; Spec ADR 0005) and objects_read_plan (its
peak memory from the header: stored plane, int32 result and
labelling_bytes; #316)
segmentation_handoff.py images_to_segment (the run's own mask pairing), segmentation_input and
resolve_segmentation/SegmentationPlan (channels by canonical ID through
resolve_request), mask_path and write_mask (a checked, compressed int32 YX TIFF,
atomic, claimed exclusively, never over an existing mask unless asked) for a segmentation
notebook; numpy and tifffile only (contract §15.5); segmentation_record.py
SegmentationRecord/SegmentationBatch and the append-only history file
<masks>/<type>/segmentation_record.json (mask type checked like a discovered one)
mask_conversion.py RunInputs.binary_to_instances / classes_to_binary (Spec ADR 0005 BM12, step
2b): new mask files from a checked mask, refused before any read or write
(check_class_image, collisions, idempotent retry); mask_lineage.py their
Expand Down
27 changes: 27 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,33 @@ old pin are refused with `spec_commit_mismatch`, as at every re-pin.

### Added

- **Segmentation hand-off** (owner request of 21 November 2026, step 1 of the Cellpose notebook
plan): `segmentation_handoff` and `segmentation_record` (numpy and tifffile only; no
segmentation tool is imported). `images_to_segment` lists the run's images that have no mask
of a type yet, `segmentation_input` reads channels **by canonical ID** through
`resolve_request` as `(C, Y, X)` stored values (the resolver's refusals unchanged, 2D only),
`write_mask` writes a checked label image as a zlib-compressed `int32` `YX` TIFF at
`<masks>/<type>/<image's folders>/<position ID>.tif`, atomically and never over an existing
mask unless asked (a masks folder inside the experiment or the output folder is refused), and
`SegmentationRecord`, `SegmentationBatch`, `write_record` and `read_record` keep one versioned,
append-only `segmentation_record.json` per mask type (a batch per run of the tool, each
written mask with its SHA-256, so a second model never erases the first's provenance); a mask
type is checked like any discovered one (`NA`, whitespace, NFC); duplicate and ambiguous masks
and a type spelled two ways are refused before writing; a mask is claimed with an exclusive create and
then moved into place, so it is never silently replaced and needs no hard links (Google Drive). The masks step skips that file silently instead of
warning `mask_without_image` (only directly in a type folder). "Has a mask" is decided by the run's own pairing (either layout, any case), `write_mask` refuses a twin elsewhere and never replaces a mask another writer made in between, and `resolve_segmentation` / `SegmentationPlan` resolve the channels once for a batch. The pairing also counts the label images inside OME-Zarr images and applies the run's own `_name_issues` and `_duplicate_issues` (no second copy of the rules), so an OME-Zarr image that has a `labels/Cells` mask is not listed and is refused by `write_mask`. `write_record` holds an exclusive lock file (`.segmentation_record.lock`, refused at once when held, removed only while it still names this writer) while it rewrites the record; it is the run's own lock, now in `lock_file.py` (`exclusive_lock`, `LockHolder`, `read_holder`) and shared with `run/lock.py`, and `new_batch_id()` makes a batch ID unique even within one second. The exclusive create and the lock protect writers on one computer only, not two Colab sessions on one Drive folder. Contract §15.5.
- **Segmentation hand-off, audit fixes (PR #367)**: the helper now applies the run's own
`_description_name_issues` to every discovered mask type (an unrelated type `NA` or a decomposed
name is refused by `images_to_segment` and `write_mask` as the next described run refuses it,
`R_MASK_TYPE_NAME_INVALID`); `write_mask` holds one exclusive per-type write lock
(`<masks>/<type>/.segmentation_write.lock`, refused at once, never waited on) from the twin check
to the published mask in both modes, so a default writer can no longer replace the finished mask
of a concurrent `overwrite=True` writer (contract §15.5 also documents the transient zero-byte
placeholder and that preprocessing must not run on the masks folder during a batch). `LockHolder`
gains a random `token`, written in the lock file and compared on removal (a lock without one,
from an older run, still reads, token `""`). `write_mask(..., on_published=callable)` calls it with the
published path while the write lock is still held (not when refused; its error propagates, the mask
stays), so a caller can hash or stamp the file before another writer can replace it.
- **Channel-identity check after the run** (contract §14.1, design
`channel-identity-check-2026-11-21` in MorphoFeature's `claude/restart-doc`): from the image
summaries the intensity checks already make (no pixel read), every imaging unit's channel is
Expand Down
55 changes: 55 additions & 0 deletions docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -1305,6 +1305,61 @@ returns can be declared, checked or reused as a run's mask, and its record has n
stated, not refused (nothing is reused in memory). **For MorphoBricks runs the run route
(§15.3, with lineage) remains the route.**

### 15.5 Segmentation hand-off (masks made by an outside tool, by channel ID)

`morphobricks_preprocessing.segmentation_handoff` and `.segmentation_record` (numpy and
tifffile only; no segmentation tool is imported or required) are what a segmentation notebook
needs from this brick. They make no decision for the tool and guess nothing.

| Function | Rule |
|---|---|
| `images_to_segment(inputs, mask_type, masks_root)` | The run's images with no mask of that type, decided by **preprocessing's own pairing**: the masks folder and the label images inside OME-Zarr images (the discovery the run and the mask step use), checked by the run's own rules (`mask_names_differ_in_case`, `duplicate_mask`), so the list and the next run cannot disagree: either layout of §15, any case in folders, name and type (`cells` is `Cells`), any TIFF extension; an OME-Zarr image with a label image of the type has its mask and is not listed. Excluded images are not in the run, so never listed. Refuses a masks folder that breaks the folder rules (§15 Folder) or that the run itself would refuse (an ambiguous mask, a link leaving the folder), and a mask type that is not one folder name. |
| `segmentation_input(inputs, channel_ids, fov, z=0, t=0)` | The channels **by canonical ID** as a `(C, Y, X)` array of the stored values, in the order of `channel_ids`. Resolved with `RunInputs.resolve_request(Request(channel_ids))` and `require()`: every refusal of the resolver passes through with its code and message (`A_SCOPE_INCOMPLETE` for a channel some images lack: choose channels every image has; `A_RGB_NOT_SUPPORTED`). Each channel is read at the C index the image stores it at (`FovSelection.index_of`), never at a position. An image with Z or T above 1 is refused: 2D only. `resolve_segmentation(inputs, channel_ids)` resolves once for the whole run and returns a `SegmentationPlan` whose `read(fov, z=0, t=0)` gives the same array; `segmentation_input` resolves on every call, so a batch uses the plan. |
| `mask_path(masks_root, mask_type, fov)` | `<masks>/<type>/<the image's folders>/<position ID>.tif` (the first layout of §15). |
| `write_mask(masks_root, mask_type, fov, labels, inputs, *, overwrite=False)` | Checks the labels first, then writes: 2D, the image's Y and X (header only), an integer or boolean type (floating point is refused even when whole), values from 0 to 2,147,483,647. Writes `int32` with zlib compression and the axes named `YX`, to a temporary file moved into place, so a reader never sees half a mask. A mask that the run's pairing finds for this image and type (any case, either layout, any extension, or a label image inside an OME-Zarr image) is refused, naming it, unless `overwrite=True`; then only the file at `mask_path` is replaced, and a twin elsewhere is still refused. Without `overwrite` the name is first claimed with an exclusive create (`O_CREAT | O_EXCL`, which fails if anything is there) and the finished file is then moved over that empty placeholder, so two writers never replace each other's mask and no hard links are needed (a Google Drive mount has none); a failed move removes the placeholder (only while empty), and a process killed in between leaves an empty file that the next write refuses as an existing mask. The masks folder may not be inside the experiment or the output folder, nor hold either (the folder rules of §15 and `masks_inside_output`). Images with Z or T above 1 are refused. |

`<masks>/<type>/segmentation_record.json` (`SegmentationRecord`, `SegmentationBatch`,
`write_record`, `read_record`, `mask_sha256`, `record_version` 1) is provenance only and an
**append-only history of batches**, so a second batch (another model, other parameters) never
erases how the first batch's masks were made. A batch holds `batch_id` (unique in the file),
`tool` and `tool_version`, `model` (`name`, `file` or null, `sha256`), `parameters` (a JSON
object), `channel_ids`, `device` (`cuda`, `cpu` or `mps`, always stated), `run_folder`,
`run_manifest_sha256`, and `images`: one entry per image (its path relative to the experiment,
once per batch), `status` `written`, `skipped` or `failed`, a `message` (required for
`failed`) and `mask_sha256`, the SHA-256 of the file written (required for `written`, null
otherwise). `new_batch_id()` makes a batch ID that is unique (the UTC start time and 8 random hex characters, so two batches started in the same second differ); an ID chosen by the caller must be unique too, since a repeated one is refused only when the record is written, after the masks. `write_record` holds an exclusive lock file (`.segmentation_record.lock`, beside the record, created with `O_CREAT | O_EXCL` and removed at the end only while it still names this writer; the run's own lock, `lock_file.exclusive_lock`) while it reads, changes and rewrites the record, and refuses at once, naming the holder, when another writer holds it (it never waits; delete the file only when no writer is running). `write_record` adds a batch; the same `batch_id` again (a batch written as it
progresses) is accepted only if everything but the images is equal and no earlier entry is
dropped or changed, otherwise it is refused and the file is left as it was.
`SegmentationRecord.latest(image)` gives the newest written entry of an image. The record is
checked on writing and on reading; an unknown version, an unknown or missing key, a value
outside its set, a batch ID twice or an image twice in a batch is refused, never repaired.
The mask type is checked with the rules the masks step applies to a discovered type
(`check_identity_text`: no surrounding whitespace, not `NA`, `nan`, `none`, `null`, `n/a` or
`<NA>`; Unicode NFC; one folder name) before anything is written. The masks step skips this
file silently when it is directly in a type folder (it is not a mask; the same name deeper
warns `mask_without_image`), and nothing reads it to decide anything.

The helper refuses what the run would refuse, before writing: an ambiguous mask or a link
leaving the folder, **any discovered mask type** that cannot be a `source_type` (the run's
`_description_name_issues`: `R_MASK_TYPE_NAME_INVALID`, a decomposed name; also for a type
other than the one asked for), a mask type spelled in two ways (`mask_names_differ_in_case`)
and an image with two masks of the type (`duplicate_mask`). A mask is never silently replaced:
without `overwrite` the name is claimed exclusively, so it works on a file system without hard
links (a Google Drive mount) too.

`write_mask` holds one exclusive write lock per mask type, `<masks>/<type>/.segmentation_write.lock`
(the run's lock file mechanism, `lock_file.exclusive_lock`), from the check for another mask of
the image to the published file, **in both modes**: an overwriting writer and a default writer
can never replace each other's mask. A held lock is refused at once, naming its holder, and
never waited on; a killed writer leaves it, and the message says to delete it. Like the
record's lock it is a dot-file, skipped by the masks step. The optional `on_published(path)` callback runs right after the publish, still under the lock (take a hash or file stamp there; it is not called when the write is refused, and an error it raises propagates with the mask left published). A lock holder carries a random
token, so a lock deleted and retaken is never removed by its earlier holder. Limits: the lock
protects the writers of one computer (not a Drive folder open in two Colab sessions); a
reader may briefly see a **zero-byte placeholder** at the mask's name while a mask is being
written (the claim before the move); and preprocessing must **not be run on the masks folder
during a batch** (a run could read that placeholder or a half-written batch). A placeholder
left by a killed writer is deleted by hand.

## 16. Shared rules for other bricks (MorphoBricks D86)

Rules every MorphoBricks brick applies in one way, so they live here as public functions
Expand Down
3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,9 @@ baseline = "tools/code_rules_baseline.json"
"morphobricks_preprocessing.masks.objects" = "pure"
"morphobricks_preprocessing.mask_rules" = "pure"
"morphobricks_preprocessing.mask_lineage" = "io"
"morphobricks_preprocessing.segmentation_handoff" = "io"
"morphobricks_preprocessing.segmentation_record" = "io"
"morphobricks_preprocessing.lock_file" = "io"
"morphobricks_preprocessing.mask_conversion" = "orchestration"
"morphobricks_preprocessing.lineage_schema" = "pure"
"morphobricks_preprocessing.code_identity" = "io"
Expand Down
32 changes: 32 additions & 0 deletions src/morphobricks_preprocessing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,24 @@
ResolvedRequest,
VolumeNotSupportedError,
)
from .segmentation_handoff import (
SegmentationPlan,
images_to_segment,
mask_path,
resolve_segmentation,
segmentation_input,
write_mask,
)
from .segmentation_record import (
SegmentationBatch,
SegmentationError,
SegmentationRecord,
SegmentedImage,
mask_sha256,
new_batch_id,
read_record,
write_record,
)
from .sources import check_sources
from .unrecorded import UnrecordedFiles, unrecorded_files
from .validation import Severity, ValidationIssue, has_errors
Expand Down Expand Up @@ -354,6 +372,11 @@
"MaskContradictsRepresentationError",
"MaskUncheckedError",
"RgbNotSupportedError",
"SegmentationBatch",
"SegmentationError",
"SegmentationPlan",
"SegmentationRecord",
"SegmentedImage",
"VolumeNotSupportedError",
"RunReportUnreadableError",
"RunStoppedError",
Expand Down Expand Up @@ -406,6 +429,9 @@
"check_sources",
"check_table",
"convert_experiment",
"images_to_segment",
"mask_path",
"mask_sha256",
"preview_conversion",
"declared_mapping",
"default_output_folder",
Expand Down Expand Up @@ -441,6 +467,7 @@
"mask_thresholds",
"named_axes",
"ngff_label_names",
"new_batch_id",
"open_ngff",
"open_ngff_labels",
"pair_counts",
Expand All @@ -459,11 +486,13 @@
"read_mask_file",
"read_physical_sizes",
"read_plane",
"read_record",
"read_sample_plane",
"reconcile_design",
"reference_axes",
"repeat_boxes",
"repeat_distributions",
"resolve_segmentation",
"review_axes",
"review_calibration",
"review_image_type",
Expand All @@ -472,13 +501,16 @@
"same_pixel_size",
"sample_planes",
"scan_masks",
"segmentation_input",
"series_axes_from_metadata",
"space_unit",
"to_micrometers",
"write_calibration",
"write_experiment_design",
"write_manifest",
"write_mask",
"write_mask_report",
"write_record",
"write_target_refusal",
"zarr_available",
]
124 changes: 124 additions & 0 deletions src/morphobricks_preprocessing/lock_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
"""An exclusive lock file: created with ``O_EXCL``, naming its holder, removed only by it.

The run's lock on its output folder (:mod:`.run.lock`) and the segmentation record's lock
(:mod:`.segmentation_record`) are both this lock. A held lock is refused at once and names
its holder; it never waits. It protects the writers of one computer: a file system shared by
several computers (a Drive folder open in two Colab sessions) does not make the exclusive
create reliable.
"""

from __future__ import annotations

import json
import os
import socket
import uuid
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import asdict, dataclass
from datetime import UTC, datetime
from pathlib import Path


@dataclass(frozen=True)
class LockHolder:
"""The process holding a lock file (a run's output folder lock or a record lock).

Attributes
----------
host
The computer it runs on.
pid
Its process ID on that computer (0 when the lock file cannot be read).
started_at
When it took the lock (UTC, ISO 8601), or empty when unknown.
token
A random ID of this one holding (``uuid4`` hex), so two holders of the same process and
second (two threads, a lock deleted and retaken) are never taken for each other; empty
for a lock file written without one (an older run lock).
"""

host: str
pid: int
started_at: str = ""
token: str = ""

def describe(self) -> str:
"""Name the holder in words, e.g. ``"process 412 on lab-pc, since 2026-11-02T10:00"``."""
if not self.pid:
return "a writer whose lock file cannot be read"
since = f", since {self.started_at}" if self.started_at else ""
return f"process {self.pid} on {self.host}{since}"


class LockHeldError(RuntimeError):
"""Raised when another writer holds a lock file.

Attributes
----------
path
The lock file.
holder
The writer holding it.
"""

def __init__(self, path: Path, holder: LockHolder) -> None:
self.path = path
self.holder = holder
super().__init__(f"{path} is locked by {holder.describe()}")


def read_holder(path: Path) -> LockHolder | None:
"""Return the holder recorded in the lock file ``path``, or None when it does not exist.

A lock file that cannot be read gives a holder with ``pid`` 0.
"""
try:
data = json.loads(path.read_text(encoding="utf-8"))
return LockHolder(
str(data["host"]),
int(data["pid"]),
str(data.get("started_at", "")),
str(data.get("token", "")),
)
except FileNotFoundError:
return None
except (OSError, ValueError, KeyError, TypeError):
return LockHolder("", 0)


@contextmanager
def exclusive_lock(path: Path) -> Iterator[LockHolder]:
"""Hold the lock file ``path`` for the block: created with ``O_CREAT | O_EXCL``.

It records this process as its holder and is removed when the block ends, also on an
error, but only while it still names this holder, token included (a lock another writer
took after a user deleted this one is never removed, even from the same process). It never
waits.

Raises
------
LockHeldError
When the file exists: another writer holds it (named by its holder).
"""
holder = LockHolder(
socket.gethostname(),
os.getpid(),
datetime.now(UTC).isoformat(timespec="seconds"),
uuid.uuid4().hex,
)
try:
handle = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
except FileExistsError:
raise LockHeldError(path, read_holder(path) or LockHolder("", 0)) from None
try:
with os.fdopen(handle, "w", encoding="utf-8") as stream:
stream.write(json.dumps(asdict(holder)) + "\n")
except BaseException:
path.unlink(missing_ok=True) # just created by this process: a lock it could not write
raise
try:
yield holder
finally:
if read_holder(path) == holder:
path.unlink(missing_ok=True)
Loading
Loading