A linear-probe benchmark for bioacoustic audio embeddings. It measures how well a frozen embedding model transfers to tasks it was not trained for: marine mammals, bats, insects, individual animals, call types, soundscape tags and environmental sounds, 15 tasks from public datasets. In the spirit of BEANS, but with grouped splits throughout (no recording, session, site or individual spans train and test), ultrasonic tasks made usable for 24-48 kHz models by time expansion, and a design that makes adding a model or a task a single small file.
probeBench downloads and prepares the data, downloads the models on first use, embeds every clip, fits one probe per task and reports the results.
One fixed recipe for every model and task:
- Each clip is loaded at the model's sample rate (time-expanded first where the task says so) and cut into windows of the model's input length with a hop of half a window; a clip shorter than a window is centered and zero-padded.
- Every window is embedded; the window embeddings are pooled per clip (mean and max, concatenated).
- Features are standardized on the training split, and a class-balanced logistic regression (one-vs-rest for multi-label tasks) is fit for each C in {0.001, 0.01, 0.1, 1, 10}; C is chosen on the validation split by the task's headline metric.
- The probe is refit on train + val with that C and scored once on the test split.
Python 3.11 or newer and the system library libsndfile
(apt-get install libsndfile1 / brew install libsndfile).
git clone https://github.com/birdnet-team/probeBench.git
cd probeBench
pip install -e ".[v3]" # BirdNET+ V3.0 (onnxruntime)
pip install -e ".[v24]" # BirdNET V2.4 (ai-edge-litert)
pip install -e ".[perch]" # Perch 2.0 (TensorFlow)
pip install -e ".[audioprotopnet]" # AudioProtoPNet-20 (torch, CUDA GPU)
pip install -e ".[birdcode]" # BirdCODE (torch, CUDA GPU, Python >= 3.12)
pip install -e ".[all]" # all threeThe base install (mel baseline, data, probe) needs none of these runtimes. The
BirdNET models run on the CPU (PROBEBENCH_THREADS threads, default 4); Perch 2.0
uses a GPU when TensorFlow sees one (PROBEBENCH_TF_GPU_MB caps its memory).
# 1. Download and prepare two small tasks (about 1.5 GB) and fetch the model
probebench setup --tasks esc50 watkins --models birdnet-v3.0
# 2. Probe the model, then the mel baseline for reference
probebench run --model birdnet-v3.0 --tasks esc50 watkins
probebench run --model mel --tasks esc50 watkins
# 3. Compare
probebench report
probebench plot --out pr.pngprobebench run --model birdnet-v3.0 without --tasks runs the whole benchmark.
Missing tasks are downloaded and prepared first (about 52 GB of downloads for all
15 tasks; archives are deleted after their extraction is verified, use
--keep-archives to keep them). probebench tasks shows what is downloaded and
prepared.
All data goes under one root: --data <dir>, else $PROBEBENCH_DATA, else
./data; model files are downloaded into <data>/models/ on first use.
| Spec | Model | Source | Runtime | Input | Embedding | License |
|---|---|---|---|---|---|---|
birdnet-v3.0 |
BirdNET+ V3.0 Global 12K, FP32 ONNX | 10.5281/zenodo.24362980 | onnxruntime | 32 kHz, 3 s | embeddings output, 1,280 |
Apache-2.0 |
birdnet-v2.4 |
BirdNET V2.4 Global 6K, FP32 TFLite | 10.5281/zenodo.15050749 | ai-edge-litert | 48 kHz, 3 s | GLOBAL_AVG_POOL tensor, 1,024 |
CC BY-NC 4.0 |
perch-v2 |
Perch 2.0 (Google DeepMind), TensorFlow SavedModel | Kaggle: google/bird-vocalization-classifier (perch_v2 v2 for GPU, perch_v2_cpu v1) |
TensorFlow | 32 kHz, 5 s | embedding output, 1,536 |
Apache-2.0 |
audioprotopnet-20 |
AudioProtoPNet-20 BirdSet XCL (ConvNeXt-base + prototype head) | Hugging Face: DBD-research-group/AudioProtoPNet-20-BirdSet-XCL | torch + transformers (GPU) | 32 kHz, 5 s | encoder feature map, global average, 1,024 | CC BY-NC 4.0 |
birdcode |
BirdCODE (Earth Species Project), ConvNeXt sound event detector | Hugging Face: EarthSpeciesProject/sed-birdcode | torch + sound-event-detection (GPU, Python >= 3.12) | 32 kHz, 5 s | encoder feature map, global average, 2,048 | MIT |
mel |
log-mel mean and standard deviation (no learning) | - | numpy / librosa | 24 kHz, 2.5 s | 256 | - |
torchscript:<file.pt> |
any TorchScript model returning (scores, embeddings) |
your file | torch (.[torch]) |
32 kHz, 3 s by default | second output | yours |
py:<module>:<factory> |
your own adapter | your code | yours | yours | yours | yours |
The BirdNET files are fetched from Zenodo with the record's MD5 checked, Perch 2.0 anonymously from Kaggle Models with the archive's size and MD5 pinned. BirdNET V2.4 is licensed CC BY-NC 4.0: non-commercial use only.
The ConvNeXt models (audioprotopnet-20, birdcode) are probed at the encoder:
the global average of the ConvNeXt feature map that their prototype or detection
head reads. Their prototype similarities and class scores are trained to their own
classes, so the encoder feature is the representation comparable to the other
models' embeddings. They need a CUDA GPU (fp16 autocast, batch
PROBEBENCH_TORCH_BATCH, default 2); their Hugging Face files go to
<data>/models/huggingface unless HF_HUB_CACHE is set. Install:
pip install -e ".[audioprotopnet]" for AudioProtoPNet,
pip install -e ".[birdcode]" for BirdCODE (installs sound-event-detection from
GitHub at a pinned commit).
Model papers:
- AudioProtoPNet: Heinrich et al. 2025, "AudioProtoPNet: An interpretable deep learning model for bird sound classification", Ecological Informatics 87, 103081, 10.1016/j.ecoinf.2025.103081.
- BirdCODE: Fine et al. 2026, "BirdCODE: Detecting bird communication at scale", bioRxiv, 10.64898/2026.07.31.742086.
- Perch 2.0: van Merriënboer et al. 2025, "Perch 2.0: The Bittern Lesson for Bioacoustics", arXiv, 10.48550/arXiv.2508.04665.
Headline metric per task on the test split (accuracy, macro-F1, macro mAP or AUC; higher is better); the mean is over all 15 tasks.
| Task | Metric | BirdNET+ V3.0 | BirdNET V2.4 | Perch 2.0 | AudioProtoPNet-20 | BirdCODE | Mel baseline |
|---|---|---|---|---|---|---|---|
| anafcc_flightcalls | macro_f1 | 0.901 | 0.767 | 0.899 | 0.927 | 0.933 | 0.486 |
| chiffchaff_acrossyear | accuracy | 0.105 | 0.175 | 0.210 | 0.095 | 0.125 | 0.175 |
| chiffchaff_withinyear | accuracy | 0.623 | 0.685 | 0.761 | 0.716 | 0.737 | 0.492 |
| chiroset_bats | macro_f1 | 0.769 | 0.835 | 0.886 | 0.895 | 0.734 | 0.647 |
| dog_individual | accuracy | 0.914 | 0.957 | 0.940 | 0.957 | 0.871 | 0.862 |
| edansa_arctic | map | 0.818 | 0.800 | 0.816 | 0.813 | 0.791 | 0.653 |
| esc50 | accuracy | 0.880 | 0.809 | 0.906 | 0.828 | 0.622 | 0.453 |
| fruit_bat_individual | accuracy | 0.758 | 0.790 | 0.815 | 0.773 | 0.767 | 0.735 |
| hainan_gibbon | auc | 0.909 | 0.766 | 0.765 | 0.919 | 0.903 | 0.801 |
| hiceas_minke | auc | 0.978 | 0.946 | 0.956 | 0.967 | 0.967 | 0.892 |
| humbugdb_species | map | 0.560 | 0.646 | 0.634 | 0.599 | 0.589 | 0.525 |
| insectset66 | accuracy | 0.923 | 0.883 | 0.937 | 0.860 | 0.737 | 0.640 |
| right_whale_upcall | auc | 0.959 | 0.913 | 0.941 | 0.967 | 0.963 | 0.897 |
| watkins | accuracy | 0.613 | 0.648 | 0.628 | 0.573 | 0.548 | 0.402 |
| zebra_finch_calltype | macro_f1 | 0.649 | 0.658 | 0.714 | 0.725 | 0.641 | 0.559 |
| Mean (15 tasks) | 0.757 | 0.752 | 0.787 | 0.774 | 0.728 | 0.615 |
The per-task numbers, chosen C values and split sizes are in results/.
Perch 2.0 was run with perch-v2 itself. The two BirdNET columns were computed
from other builds of the same models (V3.0: the TorchScript release; V2.4: its
SavedModel); on 261 real windows their embeddings match the
Zenodo files birdnet-v3.0 and birdnet-v2.4 load (cosine ≥ 0.9999 for V3.0 and
≥ 0.999 for V2.4), and a rerun of ESC-50 and Watkins with those specs reproduces
the table within 0.01 (V3.0 0.888 and 0.613, V2.4 0.809 and 0.658).
Perch 2.0 has the best mean headline metric (0.787), AudioProtoPNet-20 is second (0.774) and has the best mean average precision in the figure (0.802); BirdNET+ V3.0 and V2.4 tie (0.757 and 0.752), and BirdCODE follows (0.728). Perch 2.0 leads on individual identity and species tasks (chiffchaffs, fruit bats, ESC-50, insects), AudioProtoPNet-20 on bats, gibbons, right whales and zebra finch call types, BirdCODE on flight calls, V3.0 on Arctic soundscapes and minke whales, and V2.4 on dogs (tied), mosquitoes and marine mammal species. BirdCODE is close to the best on bird and whale calls but weaker on general sound and insects (ESC-50 0.622, InsectSet66 0.737).
| Task | Kind | Metric | What | Split grouped by | Download |
|---|---|---|---|---|---|
| watkins | multiclass | accuracy | Marine mammal species (Watkins "best of" cuts, via BEANS) | tape | 0.9 GB |
| dog_individual | multiclass | accuracy | Which of 10 dogs barks (Yin & McCowan 2004, via BEANS) | session | 0.9 GB |
| hiceas_minke | detection | auc | Minke whale boing in a 1 min HICEAS recording (via BEANS) | recorder + day | 1.4 GB |
| fruit_bat_individual | multiclass | accuracy | Which Egyptian fruit bat calls (Prat et al. 2017, via BEANS), x10 | blocks of file ids | 5.2 GB |
| zebra_finch_calltype | multiclass | macro_f1 | Zebra finch call type (Elie & Theunissen 2016) | bird | 0.2 GB |
| humbugdb_species | multiclass | map | Mosquito species in flight (HumBugDB) | place + day | 4.1 GB |
| insectset66 | multiclass | accuracy | Orthoptera and cicada species (InsectSet66) | source recording (official) | 5.2 GB |
| chiffchaff_withinyear | multiclass | accuracy | Which chiffchaff male calls, same season (Stowell et al. 2019) | official lists | 2.7 GB |
| chiffchaff_acrossyear | multiclass | accuracy | Same, tested in a later season | official lists | (shared) |
| anafcc_flightcalls | multiclass | macro_f1 | Night flight-call species (BirdVox-ANAFCC) | unit + time block | 5.7 GB |
| edansa_arctic | multilabel | map | Arctic soundscape tags (EDANSA-2019) | site (official) | 7.9 GB |
| hainan_gibbon | detection | auc | Hainan gibbon call in a 10 s window (Dufourq et al. 2021) | recording | 11.9 GB |
| chiroset_bats | multiclass | macro_f1 | European bat species (ChirosetEurope), x10 | source + place + date (official) | 9.6 GB |
| esc50 | multiclass | accuracy | Environmental sound class (ESC-50) | official folds | 0.6 GB |
| right_whale_upcall | detection | auc | Right-whale upcall in a 2 s clip (Cornell challenge, MONSTER) | none (no ids in the data) | 0.5 GB |
x10: ultrasonic recordings are time-expanded tenfold on load (the native sample rate is divided by 10), so 20-120 kHz calls land at 2-12 kHz and a 5 ms call lasts 50 ms. Results on these tasks measure how well an embedding transfers to spectro-temporal patterns it never heard at that scale, not ultrasonic hearing.
Each task's task.json records its source, license and citation. Details that
shape the numbers:
- ESC-50 drops the 680 clips whose Freesound source recording is also in FSD50K or
UrbanSound8K, which many audio models train on (list:
src/probebench/datasets/esc50_excluded.txt). - BirdVox-ANAFCC uses the BirdVox-70k, CLO and Old Bird clips only (the dataset's Xeno-canto and Macaulay files are not downloaded) and at most 1,500 clips per species.
- ChirosetEurope uses the recordings in the dataset's own
audio.zip; its link-only rows (recordings hosted on other platforms) are not fetched. - HumBugDB's headline metric is mAP: under the place/day split macro-F1 caps near 0.5 for every model.
- Licenses differ per dataset; several are non-commercial or academic-use only. The code here only downloads from the original hosts.
A model adapter is any object with name, sample_rate, window_s and
embed(x), mapping float32 windows [N, samples] to embeddings [N, D]:
# my_model.py
import numpy as np
class MyModel:
name = "my_model"
sample_rate = 32000
window_s = 3.0
def __init__(self):
self.net = ... # load or download weights here
def embed(self, x: np.ndarray) -> np.ndarray:
return self.net(x)
def make():
return MyModel()probebench run --model py:my_model:makeTo give it a short name inside the package, decorate the factory with
@register("my-model") in a module under src/probebench/models/ (see mel.py;
the factory receives the folder to cache model files in, <data>/models) and
import that module in models/__init__.py. TorchScript files that take raw audio
and return (scores, embeddings) work as they are:
--model torchscript:model.pt@my_name.
Add a module under src/probebench/datasets/ with a DATASETS list (download
locations and a prepare(raw_dir, tasks_root) function that writes task.json and
manifest.parquet through write_task) and list it in datasets/__init__.py. The
task format is documented in src/probebench/tasks.py; datasets/esc50.py is a
short example. Use grouped splits (grouped_split, per_class_group_split) so no
recording, session or individual spans train and test.
<data>/raw/<dataset>/ downloads; archives extracted to extracted/
<data>/models/zenodo_<record>/ model files, downloaded on first use
<data>/tasks/<task>/task.json kind, metric, time expansion, source, license, citation
<data>/tasks/<task>/manifest.parquet clip_id, path, start_s, end_s, label, split, group
<data>/cache/<model>/<task>.npz pooled clip embeddings (reused across runs)
<data>/results/<model>.json per task: headline, all test metrics, chosen C, split sizes
<data>/scores/<model>/<task>.npz test scores and labels (for the PR figure)
- Source code and published results: MIT License
- Datasets: not part of this repository; probeBench downloads them from their
original hosts, and each keeps its own license, recorded in its task's
task.json. Several are non-commercial or academic-use only. - Models: not part of this repository; each keeps its own license (BirdNET+ V3.0: Apache-2.0; BirdNET V2.4: CC BY-NC 4.0, non-commercial use only; Perch 2.0: Apache-2.0; AudioProtoPNet-20: CC BY-NC 4.0, non-commercial use only).
If you use probeBench, please cite it using CITATION.cff and cite
the datasets behind the tasks you report (each task's task.json lists its citation).
See CONTRIBUTING.md for guidelines. New models and tasks are welcome; AI-assisted contributions too, keep PRs focused and review every line.
BirdNET is built to support biodiversity research and conservation. See ACCEPTABLE_USE.md for how we expect the code to be used, including uses we consider unacceptable and guidance on branding derivative work.
Our work in the Cornell K. Lisa Yang Center for Conservation Bioacoustics is made possible by the generosity of K. Lisa Yang to advance innovative conservation technologies to inspire and inform the conservation of wildlife and habitats.
The development of BirdNET is supported by the German Federal Ministry of Research, Technology and Space (FKZ 01|S22072), the German Federal Ministry for the Environment, Climate Action, Nature Conservation and Nuclear Safety (FKZ 67KI31040E), the German Federal Ministry of Economic Affairs and Energy (FKZ 16KN095550), the Deutsche Bundesstiftung Umwelt (project 39263/01) and the European Social Fund.
BirdNET is a joint effort of partners from academia and industry. Without these partnerships, this project would not have been possible. Thank you!


