CounterFactual Retrieval for Evidence-Aware Individual Animal Identification

This repository contains code for CounterFactual Retrieval for Evidence-Aware Individual Animal Identification, accepted at WACV 2027. It provides mask generation, CFR evaluation, and experiment configurations for individual animal identification. Foreground masks are available on Hugging Face.

Install

Use Python 3.10 or newer in a dedicated environment:

git clone --depth 1 https://github.com/Imageomics/cfr.git
cd cfr
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

This installs the lightweight input validation, mask selection, and export package. For CFR evaluation, also install python -m pip install -r requirements.txt. For SAM3 inference, use the separate pinned environment described in mask generation. Source images and model weights are obtained separately.

Download masks

Two mask versions are available. Use the historical masks for the original evaluation; updated masks define a different experiment.

Version	Purpose	Extractor
paper	Original 34-dataset evaluation mask release	Legacy SAM3 script
standardized	Updated annotation-input selection procedure	sam3_masks

Download only the datasets you need:

python -m pip install huggingface_hub
python scripts/download_masks.py --release paper --datasets NyalaData --output-dir data/masks
python scripts/download_masks.py --release standardized --datasets NyalaData --output-dir data/masks

The downloader pins verified HF commits, checks file hashes, and separates the two versions. See mask versions and file formats for dataset lists, coordinates, and source matching. The historical code and artifacts are also preserved under the GitHub tag paper-code-v1.

CFR mask loading and CFR/SSI evaluation support is integrated into WildlifeDatasets; see the upstream integration PR for the implementation.

Generate masks

Updated extractor: supply annotation metadata, an image directory, and a species/body-part prompt. Validate inputs before running inference:

python -m sam3_masks --metadata annotations.csv --image-root images \
  --dataset MyDataset --prompt zebra --dry-run
python -m sam3_masks --metadata annotations.csv --image-root images \
  --dataset MyDataset --prompt zebra --output-dir outputs/masks --device cuda:0

Source boxes are cropped before SAM3; prepared crops and full images are used as supplied. Among valid candidates whose boxes contain the input center, selection uses mask pixel count, then confidence, then candidate order. A single annotation does not establish centered framing. Species routing, multiple prompts, optional confidence fallback, and output files are explained in the updated extraction guide.

Legacy extractor: the original script, job launcher, and prompt template remain available. Its box-IoU/highest-confidence rules differ from the updated extractor. Follow legacy extraction; downloading the historical masks is the direct way to use the frozen release.

Evaluate CFR

The default checkpoints are MIEW-ID MSV3, loaded through Transformers, and MegaDescriptor-L-384, loaded through timm. During embedding extraction, these libraries download and cache the checkpoints when needed. MIEW-ID also loads the provider's custom model code with trust_remote_code=True. The model cards provide training details and citations.

Obtain the source images and matching identity/crop metadata, then configure their paths:

cp configs/cfr_dataset_index.template.tsv configs/cfr_dataset_index.tsv

Edit the index to include your datasets, image roots, metadata, and selected mask-version paths. Annotation IDs and image/crop coordinates must agree; downloading masks alone does not supply identity labels. See dataset acquisition and evaluation inputs.

python scripts/run_cfr_evaluation.py \
  --central_index configs/cfr_dataset_index.tsv --suite all \
  --models miewid megadescriptor --mode extract_eval \
  --output_root outputs/cfr --ks 1 5 10 --device cuda:0

Evaluation writes JSON/CSV metrics and local embedding caches. The reference protocol uses accepted-mask annotations with known identities, excludes the query itself from the gallery, and leaves same-image and same-encounter exclusions off. CFR/SSI definitions and silhouette settings are documented with the evaluation inputs.

Additional experiments

• Mask perturbations and image-corruption robustness.
• Foreground-mixed training checkpoint evaluation.
• Optional cluster launchers in slurm/ use editable account, partition, and path placeholders.

Run the lightweight regression suite with python -m unittest discover -s tests -v. It checks selection, input handling, exports, fallback, and mask downloads without downloading model weights. GPU inference and retrieval require the separate model dependencies and source data.

Citation

Machine-readable software citation metadata is available in CITATION.cff, with matching archive metadata in .zenodo.json.

The software is archived on Zenodo. Cite the all-versions DOI to reference CFR across releases.

Please cite CFR and the relevant source datasets/papers, WildlifeDatasets, and SAM3.

@inproceedings{upadhyay2027cfr,
  author = {Ankit Kumar Upadhyay and Aleksandr Algasov and Ekaterina Nepovinnykh
            and Fedor Zolotarev and Tuomas Eerola and Heikki Kälviäinen
            and Charles V. Stewart and Lukáš Picek},
  title = {CounterFactual Retrieval for Evidence-Aware Individual Animal Identification},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year = {2027},
  note = {Accepted}
}

License

CFR code is licensed under MIT. Released masks are licensed under CC BY 4.0. Source datasets, SAM3, model weights, and other dependencies retain their respective licenses and terms.

Acknowledgements

Supported by the Imageomics Institute, funded by the US National Science Foundation's Harnessing the Data Revolution program under Award #2118240. Opinions, findings, conclusions and recommendations are those of the authors and do not necessarily reflect the views of the National Science Foundation.