Skip to content

About

A linear-probe benchmark for bioacoustic audio embeddings

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

probeBench

BirdNET
Code license: MIT Python 3.11+ CI 15 tasks

A linear-probe benchmark for bioacoustic audio embeddings. It measures how well a frozen embedding model transfers to tasks it was not trained for: marine mammals, bats, insects, individual animals, call types, soundscape tags and environmental sounds, 15 tasks from public datasets. In the spirit of BEANS, but with grouped splits throughout (no recording, session, site or individual spans train and test), ultrasonic tasks made usable for 24-48 kHz models by time expansion, and a design that makes adding a model or a task a single small file.

probeBench downloads and prepares the data, downloads the models on first use, embeds every clip, fits one probe per task and reports the results.

The recipe

One fixed recipe for every model and task:

  1. Each clip is loaded at the model's sample rate (time-expanded first where the task says so) and cut into windows of the model's input length with a hop of half a window; a clip shorter than a window is centered and zero-padded.
  2. Every window is embedded; the window embeddings are pooled per clip (mean and max, concatenated).
  3. Features are standardized on the training split, and a class-balanced logistic regression (one-vs-rest for multi-label tasks) is fit for each C in {0.001, 0.01, 0.1, 1, 10}; C is chosen on the validation split by the task's headline metric.
  4. The probe is refit on train + val with that C and scored once on the test split.

Install

Python 3.11 or newer and the system library libsndfile (apt-get install libsndfile1 / brew install libsndfile).

git clone https://github.com/birdnet-team/probeBench.git
cd probeBench
pip install -e ".[v3]"       # BirdNET+ V3.0 (onnxruntime)
pip install -e ".[v24]"      # BirdNET V2.4 (ai-edge-litert)
pip install -e ".[perch]"    # Perch 2.0 (TensorFlow)
pip install -e ".[audioprotopnet]"  # AudioProtoPNet-20 (torch, CUDA GPU)
pip install -e ".[birdcode]" # BirdCODE (torch, CUDA GPU, Python >= 3.12)
pip install -e ".[all]"      # all three

The base install (mel baseline, data, probe) needs none of these runtimes. The BirdNET models run on the CPU (PROBEBENCH_THREADS threads, default 4); Perch 2.0 uses a GPU when TensorFlow sees one (PROBEBENCH_TF_GPU_MB caps its memory).

Quickstart

# 1. Download and prepare two small tasks (about 1.5 GB) and fetch the model
probebench setup --tasks esc50 watkins --models birdnet-v3.0

# 2. Probe the model, then the mel baseline for reference
probebench run --model birdnet-v3.0 --tasks esc50 watkins
probebench run --model mel --tasks esc50 watkins

# 3. Compare
probebench report
probebench plot --out pr.png

probebench run --model birdnet-v3.0 without --tasks runs the whole benchmark. Missing tasks are downloaded and prepared first (about 52 GB of downloads for all 15 tasks; archives are deleted after their extraction is verified, use --keep-archives to keep them). probebench tasks shows what is downloaded and prepared.

All data goes under one root: --data <dir>, else $PROBEBENCH_DATA, else ./data; model files are downloaded into <data>/models/ on first use.

Models

Spec Model Source Runtime Input Embedding License
birdnet-v3.0 BirdNET+ V3.0 Global 12K, FP32 ONNX 10.5281/zenodo.24362980 onnxruntime 32 kHz, 3 s embeddings output, 1,280 Apache-2.0
birdnet-v2.4 BirdNET V2.4 Global 6K, FP32 TFLite 10.5281/zenodo.15050749 ai-edge-litert 48 kHz, 3 s GLOBAL_AVG_POOL tensor, 1,024 CC BY-NC 4.0
perch-v2 Perch 2.0 (Google DeepMind), TensorFlow SavedModel Kaggle: google/bird-vocalization-classifier (perch_v2 v2 for GPU, perch_v2_cpu v1) TensorFlow 32 kHz, 5 s embedding output, 1,536 Apache-2.0
audioprotopnet-20 AudioProtoPNet-20 BirdSet XCL (ConvNeXt-base + prototype head) Hugging Face: DBD-research-group/AudioProtoPNet-20-BirdSet-XCL torch + transformers (GPU) 32 kHz, 5 s encoder feature map, global average, 1,024 CC BY-NC 4.0
birdcode BirdCODE (Earth Species Project), ConvNeXt sound event detector Hugging Face: EarthSpeciesProject/sed-birdcode torch + sound-event-detection (GPU, Python >= 3.12) 32 kHz, 5 s encoder feature map, global average, 2,048 MIT
mel log-mel mean and standard deviation (no learning) - numpy / librosa 24 kHz, 2.5 s 256 -
torchscript:<file.pt> any TorchScript model returning (scores, embeddings) your file torch (.[torch]) 32 kHz, 3 s by default second output yours
py:<module>:<factory> your own adapter your code yours yours yours yours

The BirdNET files are fetched from Zenodo with the record's MD5 checked, Perch 2.0 anonymously from Kaggle Models with the archive's size and MD5 pinned. BirdNET V2.4 is licensed CC BY-NC 4.0: non-commercial use only.

The ConvNeXt models (audioprotopnet-20, birdcode) are probed at the encoder: the global average of the ConvNeXt feature map that their prototype or detection head reads. Their prototype similarities and class scores are trained to their own classes, so the encoder feature is the representation comparable to the other models' embeddings. They need a CUDA GPU (fp16 autocast, batch PROBEBENCH_TORCH_BATCH, default 2); their Hugging Face files go to <data>/models/huggingface unless HF_HUB_CACHE is set. Install: pip install -e ".[audioprotopnet]" for AudioProtoPNet, pip install -e ".[birdcode]" for BirdCODE (installs sound-event-detection from GitHub at a pinned commit).

Model papers:

  • AudioProtoPNet: Heinrich et al. 2025, "AudioProtoPNet: An interpretable deep learning model for bird sound classification", Ecological Informatics 87, 103081, 10.1016/j.ecoinf.2025.103081.
  • BirdCODE: Fine et al. 2026, "BirdCODE: Detecting bird communication at scale", bioRxiv, 10.64898/2026.07.31.742086.
  • Perch 2.0: van Merriënboer et al. 2025, "Perch 2.0: The Bittern Lesson for Bioacoustics", arXiv, 10.48550/arXiv.2508.04665.

Results

Headline metric per task on the test split (accuracy, macro-F1, macro mAP or AUC; higher is better); the mean is over all 15 tasks.

Task Metric BirdNET+ V3.0 BirdNET V2.4 Perch 2.0 AudioProtoPNet-20 BirdCODE Mel baseline
anafcc_flightcalls macro_f1 0.901 0.767 0.899 0.927 0.933 0.486
chiffchaff_acrossyear accuracy 0.105 0.175 0.210 0.095 0.125 0.175
chiffchaff_withinyear accuracy 0.623 0.685 0.761 0.716 0.737 0.492
chiroset_bats macro_f1 0.769 0.835 0.886 0.895 0.734 0.647
dog_individual accuracy 0.914 0.957 0.940 0.957 0.871 0.862
edansa_arctic map 0.818 0.800 0.816 0.813 0.791 0.653
esc50 accuracy 0.880 0.809 0.906 0.828 0.622 0.453
fruit_bat_individual accuracy 0.758 0.790 0.815 0.773 0.767 0.735
hainan_gibbon auc 0.909 0.766 0.765 0.919 0.903 0.801
hiceas_minke auc 0.978 0.946 0.956 0.967 0.967 0.892
humbugdb_species map 0.560 0.646 0.634 0.599 0.589 0.525
insectset66 accuracy 0.923 0.883 0.937 0.860 0.737 0.640
right_whale_upcall auc 0.959 0.913 0.941 0.967 0.963 0.897
watkins accuracy 0.613 0.648 0.628 0.573 0.548 0.402
zebra_finch_calltype macro_f1 0.649 0.658 0.714 0.725 0.641 0.559
Mean (15 tasks) 0.757 0.752 0.787 0.774 0.728 0.615

Mean and per-task precision-recall curves

The per-task numbers, chosen C values and split sizes are in results/. Perch 2.0 was run with perch-v2 itself. The two BirdNET columns were computed from other builds of the same models (V3.0: the TorchScript release; V2.4: its SavedModel); on 261 real windows their embeddings match the Zenodo files birdnet-v3.0 and birdnet-v2.4 load (cosine ≥ 0.9999 for V3.0 and ≥ 0.999 for V2.4), and a rerun of ESC-50 and Watkins with those specs reproduces the table within 0.01 (V3.0 0.888 and 0.613, V2.4 0.809 and 0.658).

Perch 2.0 has the best mean headline metric (0.787), AudioProtoPNet-20 is second (0.774) and has the best mean average precision in the figure (0.802); BirdNET+ V3.0 and V2.4 tie (0.757 and 0.752), and BirdCODE follows (0.728). Perch 2.0 leads on individual identity and species tasks (chiffchaffs, fruit bats, ESC-50, insects), AudioProtoPNet-20 on bats, gibbons, right whales and zebra finch call types, BirdCODE on flight calls, V3.0 on Arctic soundscapes and minke whales, and V2.4 on dogs (tied), mosquitoes and marine mammal species. BirdCODE is close to the best on bird and whale calls but weaker on general sound and insects (ESC-50 0.622, InsectSet66 0.737).

Tasks

Task Kind Metric What Split grouped by Download
watkins multiclass accuracy Marine mammal species (Watkins "best of" cuts, via BEANS) tape 0.9 GB
dog_individual multiclass accuracy Which of 10 dogs barks (Yin & McCowan 2004, via BEANS) session 0.9 GB
hiceas_minke detection auc Minke whale boing in a 1 min HICEAS recording (via BEANS) recorder + day 1.4 GB
fruit_bat_individual multiclass accuracy Which Egyptian fruit bat calls (Prat et al. 2017, via BEANS), x10 blocks of file ids 5.2 GB
zebra_finch_calltype multiclass macro_f1 Zebra finch call type (Elie & Theunissen 2016) bird 0.2 GB
humbugdb_species multiclass map Mosquito species in flight (HumBugDB) place + day 4.1 GB
insectset66 multiclass accuracy Orthoptera and cicada species (InsectSet66) source recording (official) 5.2 GB
chiffchaff_withinyear multiclass accuracy Which chiffchaff male calls, same season (Stowell et al. 2019) official lists 2.7 GB
chiffchaff_acrossyear multiclass accuracy Same, tested in a later season official lists (shared)
anafcc_flightcalls multiclass macro_f1 Night flight-call species (BirdVox-ANAFCC) unit + time block 5.7 GB
edansa_arctic multilabel map Arctic soundscape tags (EDANSA-2019) site (official) 7.9 GB
hainan_gibbon detection auc Hainan gibbon call in a 10 s window (Dufourq et al. 2021) recording 11.9 GB
chiroset_bats multiclass macro_f1 European bat species (ChirosetEurope), x10 source + place + date (official) 9.6 GB
esc50 multiclass accuracy Environmental sound class (ESC-50) official folds 0.6 GB
right_whale_upcall detection auc Right-whale upcall in a 2 s clip (Cornell challenge, MONSTER) none (no ids in the data) 0.5 GB

x10: ultrasonic recordings are time-expanded tenfold on load (the native sample rate is divided by 10), so 20-120 kHz calls land at 2-12 kHz and a 5 ms call lasts 50 ms. Results on these tasks measure how well an embedding transfers to spectro-temporal patterns it never heard at that scale, not ultrasonic hearing.

Each task's task.json records its source, license and citation. Details that shape the numbers:

  • ESC-50 drops the 680 clips whose Freesound source recording is also in FSD50K or UrbanSound8K, which many audio models train on (list: src/probebench/datasets/esc50_excluded.txt).
  • BirdVox-ANAFCC uses the BirdVox-70k, CLO and Old Bird clips only (the dataset's Xeno-canto and Macaulay files are not downloaded) and at most 1,500 clips per species.
  • ChirosetEurope uses the recordings in the dataset's own audio.zip; its link-only rows (recordings hosted on other platforms) are not fetched.
  • HumBugDB's headline metric is mAP: under the place/day split macro-F1 caps near 0.5 for every model.
  • Licenses differ per dataset; several are non-commercial or academic-use only. The code here only downloads from the original hosts.

Adding a model

A model adapter is any object with name, sample_rate, window_s and embed(x), mapping float32 windows [N, samples] to embeddings [N, D]:

# my_model.py
import numpy as np


class MyModel:
    name = "my_model"
    sample_rate = 32000
    window_s = 3.0

    def __init__(self):
        self.net = ...  # load or download weights here

    def embed(self, x: np.ndarray) -> np.ndarray:
        return self.net(x)


def make():
    return MyModel()
probebench run --model py:my_model:make

To give it a short name inside the package, decorate the factory with @register("my-model") in a module under src/probebench/models/ (see mel.py; the factory receives the folder to cache model files in, <data>/models) and import that module in models/__init__.py. TorchScript files that take raw audio and return (scores, embeddings) work as they are: --model torchscript:model.pt@my_name.

Adding a task

Add a module under src/probebench/datasets/ with a DATASETS list (download locations and a prepare(raw_dir, tasks_root) function that writes task.json and manifest.parquet through write_task) and list it in datasets/__init__.py. The task format is documented in src/probebench/tasks.py; datasets/esc50.py is a short example. Use grouped splits (grouped_split, per_class_group_split) so no recording, session or individual spans train and test.

Data layout

<data>/raw/<dataset>/             downloads; archives extracted to extracted/
<data>/models/zenodo_<record>/    model files, downloaded on first use
<data>/tasks/<task>/task.json     kind, metric, time expansion, source, license, citation
<data>/tasks/<task>/manifest.parquet  clip_id, path, start_s, end_s, label, split, group
<data>/cache/<model>/<task>.npz   pooled clip embeddings (reused across runs)
<data>/results/<model>.json       per task: headline, all test metrics, chosen C, split sizes
<data>/scores/<model>/<task>.npz  test scores and labels (for the PR figure)

License

  • Source code and published results: MIT License
  • Datasets: not part of this repository; probeBench downloads them from their original hosts, and each keeps its own license, recorded in its task's task.json. Several are non-commercial or academic-use only.
  • Models: not part of this repository; each keeps its own license (BirdNET+ V3.0: Apache-2.0; BirdNET V2.4: CC BY-NC 4.0, non-commercial use only; Perch 2.0: Apache-2.0; AudioProtoPNet-20: CC BY-NC 4.0, non-commercial use only).

Citation

If you use probeBench, please cite it using CITATION.cff and cite the datasets behind the tasks you report (each task's task.json lists its citation).

Contributing

See CONTRIBUTING.md for guidelines. New models and tasks are welcome; AI-assisted contributions too, keep PRs focused and review every line.

Acceptable Use

BirdNET is built to support biodiversity research and conservation. See ACCEPTABLE_USE.md for how we expect the code to be used, including uses we consider unacceptable and guidance on branding derivative work.

Funding

Our work in the Cornell K. Lisa Yang Center for Conservation Bioacoustics is made possible by the generosity of K. Lisa Yang to advance innovative conservation technologies to inspire and inform the conservation of wildlife and habitats.

The development of BirdNET is supported by the German Federal Ministry of Research, Technology and Space (FKZ 01|S22072), the German Federal Ministry for the Environment, Climate Action, Nature Conservation and Nuclear Safety (FKZ 67KI31040E), the German Federal Ministry of Economic Affairs and Energy (FKZ 16KN095550), the Deutsche Bundesstiftung Umwelt (project 39263/01) and the European Social Fund.

Partners

BirdNET is a joint effort of partners from academia and industry. Without these partnerships, this project would not have been possible. Thank you!

Logos of all partners

About

A linear-probe benchmark for bioacoustic audio embeddings

Resources

Code of conduct

Contributing

Security policy

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages