Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,23 @@ old pin are refused with `spec_commit_mismatch`, as at every re-pin.
from an older run, still reads, token `""`). `write_mask(..., on_published=callable)` calls it with the
published path while the write lock is still held (not when refused; its error propagates, the mask
stays), so a caller can hash or stamp the file before another writer can replace it.
- **`write_mask` rolls back when `on_published` fails** (external audit of #375, 22 November 2026).
The caller's provenance step now runs in the callback under the type's write lock; if it raises
(any exception, `KeyboardInterrupt` too), `write_mask` restores an overwritten mask byte for byte
from a hidden copy kept beside it (`.<name>.previous.<random>`, skipped by the masks step, removed
on success) or removes a new mask, still under the lock, then re-raises. A rollback that cannot
restore or remove raises `SegmentationError` naming both files and keeps the copy. Before, a
failed overwrite left the new mask published and the earlier valid mask was lost. The record's
duplicate-image check is linear (a seen-set instead of `list.count` per entry).
- **Docs: `on_published` rollback and record durability** (external audit, 22 November 2026): the
`write_mask` docstring and contract §15.5 say that the rollback happens on every callback exception
even after the callback committed (it must reconcile its own post-commit failures), and that the
record is replaced atomically but not fsynced (not a power-loss guarantee). Behaviour unchanged.
- **`write_mask` decides its rollback from the file system** (review of #397): an interrupt right
after the move but before `write_mask` returned used to delete the copy of the earlier mask, so the
new mask stayed and the earlier one was lost; now the target's inode says whether the move happened
and the earlier mask is restored (a new mask removed). New `stray_backups(masks_root, mask_type)`
lists `.<name>.previous.<random>` copies left by killed processes; they are reported, never deleted.
- **Channel-identity check after the run** (contract §14.1, design
`channel-identity-check-2026-11-21` in MorphoFeature's `claude/restart-doc`): from the image
summaries the intensity checks already make (no pixel read), every imaging unit's channel is
Expand Down
2 changes: 1 addition & 1 deletion docs/contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -1352,7 +1352,7 @@ links (a Google Drive mount) too.
the image to the published file, **in both modes**: an overwriting writer and a default writer
can never replace each other's mask. A held lock is refused at once, naming its holder, and
never waited on; a killed writer leaves it, and the message says to delete it. Like the
record's lock it is a dot-file, skipped by the masks step. The optional `on_published(path)` callback runs right after the publish, still under the lock (take a hash or file stamp there; it is not called when the write is refused, and an error it raises propagates with the mask left published). A lock holder carries a random
record's lock it is a dot-file, skipped by the masks step. The optional `on_published(path)` callback runs right after the publish, still under the lock (take a hash and write the provenance there; it is not called when the write is refused). If it raises (any exception, `KeyboardInterrupt` too), the write is **rolled back under the same lock** and the error propagates: with `overwrite=True` the earlier mask was first copied to a hidden file beside it (`.<name>.previous.<random>`, a dot-file the masks step skips, removed once the callback returns) and is moved back over the new one byte for byte, with its mode; a new mask is removed. A rollback that cannot restore or remove is a `SegmentationError` naming both files, and the copy is kept. The rollback is unconditional: it happens on **every** exception from the callback, even one raised after the callback had durably committed its provenance, so a callback that commits must reconcile its own post-commit failures (keep the mask and not raise, or undo its commit) before it raises; `write_mask` has no committed-state protocol. The provenance record is written atomically (a temporary file replaced into place) but **not fsynced**: that protects readers from a half-written record, not a power loss, after which a record or mask may be missing or stale. A lock holder carries a random
token, so a lock deleted and retaken is never removed by its earlier holder. Limits: the lock
protects the writers of one computer (not a Drive folder open in two Colab sessions); a
reader may briefly see a **zero-byte placeholder** at the mask's name while a mask is being
Expand Down
2 changes: 2 additions & 0 deletions src/morphobricks_preprocessing/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,7 @@
mask_path,
resolve_segmentation,
segmentation_input,
stray_backups,
write_mask,
)
from .segmentation_record import (
Expand Down Expand Up @@ -502,6 +503,7 @@
"sample_planes",
"scan_masks",
"segmentation_input",
"stray_backups",
"series_axes_from_metadata",
"space_unit",
"to_micrometers",
Expand Down
151 changes: 128 additions & 23 deletions src/morphobricks_preprocessing/segmentation_handoff.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
imports no segmentation tool (contract §15.5):

- :func:`images_to_segment`: the run's images that have no mask of a mask type yet;
- :func:`stray_backups`: hidden copies of earlier masks left by a killed overwrite (reported,
never deleted);
- :func:`segmentation_input`: the channels asked for **by canonical channel ID**, as a
``(C, Y, X)`` array of the stored values in the order asked, never by position;
- :func:`mask_path` and :func:`write_mask`: one label image per image at
Expand All @@ -23,6 +25,7 @@
from __future__ import annotations

import os
import shutil
import tempfile
from collections.abc import Callable
from dataclasses import dataclass
Expand Down Expand Up @@ -241,8 +244,15 @@ def write_mask(
on_published
Called with the published mask's path right after it is published, while the type's
write lock is still held (so the file cannot be replaced meanwhile: hash or stamp it
here). Not called when the write is refused. An error it raises propagates; the mask
stays published and the caller decides what to do with it.
here, and write its provenance). Not called when the write is refused. If it raises
(any exception, ``KeyboardInterrupt`` too), the write is rolled back under the lock:
an overwritten mask is restored byte for byte from a hidden copy kept beside it
(``.<name>.previous.<random>``, removed once the callback returns), a new mask is
removed; then the error propagates. The rollback happens on EVERY exception, even if the
callback had already durably committed its provenance: a callback that commits must
reconcile its own post-commit failures (and not raise) before returning or raising.
The provenance record is written atomically (replaced) but not fsynced, so a power
loss can lose it: this is not a durability guarantee.

Returns
-------
Expand All @@ -257,7 +267,7 @@ def write_mask(
without ``overwrite`` (or another TIFF of the same name); a masks folder inside the
experiment or the output folder; or another writer holding the type's write lock
(``<masks>/<type>/.segmentation_write.lock``), which is refused at once and names the
holder.
holder; or a rollback that could not restore or remove the mask (naming the files).
"""
target = mask_path(masks_root, mask_type, fov)
_check_folders(inputs, masks_root)
Expand All @@ -269,27 +279,122 @@ def write_mask(
# one writer per type for the twin check, the claim and the publish, in both modes
_refuse_twins(inputs, masks_root, mask_type, fov, target, overwrite)
target.parent.mkdir(parents=True, exist_ok=True)
handle, temporary = tempfile.mkstemp(
prefix=f".{target.name}.", suffix=".tmp", dir=target.parent
)
os.close(handle)
try:
tifffile.imwrite(
temporary,
data,
photometric="minisblack",
compression="zlib",
metadata={"axes": "YX"},
)
give_output_mode(temporary, target)
_publish(
Path(temporary), target, overwrite, f"The {mask_type} mask of {fov.relative_path}"
)
except BaseException:
Path(temporary).unlink(missing_ok=True)
raise
what = f"The {mask_type} mask of {fov.relative_path}"
backup = _keep_backup(target) if overwrite and target.is_file() else None
_write_and_publish(data, target, overwrite, what, backup) # rolls itself back on failure
if on_published is not None:
on_published(target) # still under the lock; an error propagates, the mask stays
try:
on_published(target) # still under the lock
except BaseException:
_roll_back(target, backup) # still under the lock; loud if it cannot
raise
_drop_backup(backup, target)
return target


def _write_and_publish(
data: np.ndarray, target: Path, overwrite: bool, what: str, backup: Path | None
) -> None:
"""Write ``data`` to a temporary file beside ``target``, then publish it there.

On any failure (``KeyboardInterrupt`` too) whether the publish happened is read from the
file system, not from whether the call returned: if ``target`` is now the temporary file
(same inode) the earlier mask is restored (or the new one removed); otherwise the backup
is dropped, the earlier mask never left. The backup is kept until that is decided.
"""
handle, temporary = tempfile.mkstemp(
prefix=f".{target.name}.", suffix=".tmp", dir=target.parent
)
os.close(handle)
inode = None
try:
tifffile.imwrite(
temporary, data, photometric="minisblack", compression="zlib", metadata={"axes": "YX"}
)
give_output_mode(temporary, target)
inode = os.stat(temporary).st_ino
_publish(Path(temporary), target, overwrite, what)
except BaseException:
Path(temporary).unlink(missing_ok=True)
if inode is not None and _is_inode(target, inode):
_roll_back(target, backup) # published, then interrupted before returning
else:
_drop_backup(backup, target) # nothing was replaced: the earlier mask is at the target
raise


def _is_inode(path: Path, inode: int) -> bool:
"""Whether ``path`` exists and is the file with this inode."""
try:
return path.stat().st_ino == inode
except OSError:
return False


def stray_backups(masks_root: str | os.PathLike, mask_type: str) -> tuple[Path, ...]:
"""Return the hidden ``.<name>.previous.<random>`` copies left in the type's folder.

A process killed while overwriting a mask leaves its copy of the earlier mask. They are
only reported, never deleted: the person decides (the earlier mask may be only there).
"""
check_mask_type(mask_type)
folder = Path(masks_root, mask_type)
return tuple(sorted(folder.rglob(".*.previous.*"))) if folder.is_dir() else ()


def _keep_backup(target: Path) -> Path:
"""Copy the mask at ``target`` to a hidden file beside it and return the copy's path.

A copy, not a move, so the earlier mask stays at its name until the new one replaces it in
one move (a process killed meanwhile leaves the earlier mask and a stray dot-file, which
the masks step skips). ``.<name>.previous.<random>``, the temporary-name pattern.
"""
handle, backup = tempfile.mkstemp(prefix=f".{target.name}.previous.", dir=target.parent)
os.close(handle)
try:
shutil.copyfile(target, backup)
give_output_mode(backup, target) # restored with the earlier mask's mode
except BaseException:
Path(backup).unlink(missing_ok=True)
raise
return Path(backup)


def _drop_backup(backup: Path | None, target: Path) -> None:
"""Remove the hidden copy of the earlier mask once ``target`` holds a mask to keep."""
if backup is None:
return
try:
backup.unlink(missing_ok=True)
except OSError as error:
raise SegmentationError(
f"{target} is in place, but the copy of the earlier mask, {backup}, could not be "
f"removed ({error}); delete it by hand."
) from error


def _roll_back(target: Path, backup: Path | None) -> None:
"""Undo a publish whose caller failed: put ``backup`` back at ``target``, or remove it.

Called under the type's write lock. A failure is raised as a SegmentationError naming the
files, never ignored; the backup is then kept.
"""
try:
if backup is None:
target.unlink(missing_ok=True)
else:
os.replace(backup, target)
except OSError as error:
if backup is None:
raise SegmentationError(
f"The new mask {target} could not be removed after its caller failed ({error}); "
"delete it by hand: it has no provenance."
) from error
raise SegmentationError(
f"The earlier mask could not be restored from {backup} to {target} after the "
f"caller failed ({error}); {target} holds the new mask, which has no provenance. "
f"Move {backup} to {target} by hand."
) from error
return target


Expand Down
7 changes: 5 additions & 2 deletions src/morphobricks_preprocessing/segmentation_record.py
Original file line number Diff line number Diff line change
Expand Up @@ -306,8 +306,11 @@ def _batch(data: object) -> dict[str, Any]:
raise SegmentationError("images must be a list.")
for entry in data["images"]:
_image(entry)
images = [entry["image"] for entry in data["images"]]
twice = sorted({name for name in images if images.count(name) > 1})
seen: set[str] = set()
repeated: set[str] = set()
for entry in data["images"]:
(repeated if entry["image"] in seen else seen).add(entry["image"])
twice = sorted(repeated)
if twice:
raise SegmentationError(
f"The batch {data['batch_id']!r} has more than one entry for {', '.join(twice)}; "
Expand Down
Loading
Loading