Key Takeaways:
- Existing bioacoustic models are largely trained and evaluated on a narrow range of tasks, particularly species and call-type recognition, leaving many capabilities relevant to animal behavior and communication underexplored.
- We introduce BEANS-Next, a benchmark grounded in a taxonomy of bioacoustics tasks spanning acoustic perception, biological category recognition, scene understanding, and in-context learning.
- We also introduce ROOTS, a training resource containing 44 million audio-language pairs from 6.5 million distinct audio clips, built using existing bioacoustic data, metadata, language-model-generated text, and synthetic audio.
- Fine-tuning NatureLM-audio with ROOTS substantially improved performance across all four tiers, including acoustic understanding, scene interpretation, and recognizing unfamiliar sounds from audio examples.
- We have open-sourced BEANS-Next and ROOTS to support more versatile bioacoustic models. This research also informs the upcoming release of NatureLM-audio v2.0.
The Gaps in Bioacoustic Model Training and Evaluation
Our earlier work on NatureLM-audio demonstrated that a single audio-language model could perform multiple bioacoustic tasks, including species identification, call-type classification, and life-stage prediction. It also demonstrated promising zero-shot generalization to species unseen during training.
However, bioacoustic training datasets and benchmarks still focus heavily on biological category recognition, particularly species and call-type classification. While these capabilities are valuable, they represent only a fraction of the questions researchers might want to ask of a recording.
A further limitation lies in the training data itself. Bioacoustic archives contain vast collections of recordings, but many are only loosely labeled or accompanied by information stored in field notes and other unstructured formats. Turning these recordings and annotations into training examples for more complex bioacoustic tasks requires substantial processing.
As a result, researchers lack both the benchmarks needed to evaluate these broader capabilities and the training resources to develop them. To address these limitations, we developed BEANS-Next, a new benchmark, and ROOTS, a large-scale audio-language training dataset.
BEANS-Next: A Benchmark Covering Four Broader Areas of Bioacoustic Capability
BEANS-Next is a new benchmark designed to evaluate a broader range of audio-language abilities relevant to bioacoustic research. It builds on previous benchmarking efforts and organizes bioacoustic tasks into four tiers: acoustic perception, biological category recognition, scene understanding, and in-context learning.
For researchers working with recordings containing multiple species, overlapping vocalizations, or unfamiliar call types, identifying a species is often only one part of the analysis. Understanding when sounds occur, how they relate to one another, and how their acoustic properties differ can be equally important.
Our evaluation of existing bioacoustic models on BEANS-Next revealed that they performed competitively on familiar recognition tasks but struggled with many of the broader capabilities. Addressing these limitations requires training data covering a wider range of bioacoustic tasks. These findings helped guide the development of ROOTS.
ROOTS: Building the Training Data These Capabilities Need
To address the training bottleneck, we developed ROOTS, a large-scale training dataset containing 44 million audio-language pairs from 6.5 million distinct audio clips. ROOTS makes more extensive use of existing bioacoustic recordings and annotations, while also introducing new ways to generate training examples across all four tiers of BEANS-Next.
We used four complementary approaches to construct the dataset:
- Curated bioacoustic archives: We brought together recordings and annotations from existing bioacoustic collections, converting information from different sources into a consistent format for model training.
- Underused and audio-derived metadata: We incorporated information about species, taxonomy, call types, life stages, and behavior, alongside acoustic measurements and descriptions derived from recordings.
- LLM-based text synthesis: We used language models to turn field notes and other unstructured annotations into natural-language questions and answers to build complex scene understanding tasks, and to generate descriptions of vocalization characteristics from spectrograms. Generated outputs were filtered and checked for unsupported or inaccurate information.
- Synthetic audio pipelines: We created synthetic recordings using existing animal sounds, with precise labels and timestamps for individual events. These provided training examples for tasks including sound detection, counting, temporal reasoning, and recognizing sounds from audio examples.
These methods allowed us to use more of the information already available in bioacoustic archives while creating training examples for tasks that are difficult to annotate at scale.
Figure 1: Overview of BEANS-Next and ROOTS, their underlying data sources, and the shared four-tier task taxonomy that grounds them.
What We Learned from Fine-Tuning NatureLM-audio
To test whether ROOTS could improve model performance, we used NatureLM-audio v1.1 as a starting point and fine-tuned a separate model with training data aligned to each of the four BEANS-Next tiers. This allowed us to examine how targeted training data contributed to different bioacoustic capabilities.
The results showed improvements across all four tiers, including tasks where existing models had previously struggled.
Tier 1: Acoustic perception
Training with ROOTS improved the model’s ability to describe and measure acoustic characteristics, including fundamental frequency and signal-to-noise ratio. For example, in a multiple-choice question task with vocalization descriptions sourced from field notes, accuracy increased from 28.7% to 97.9%. Improvements in fundamental frequency estimation also extended to species held out from training, suggesting that some of these acoustic capabilities can generalize beyond the species represented in the training data.
Tier 2: Biological category recognition
Performance improved on tasks involving species presence, call types, and behavior prediction. For example, accuracy on a behavior prediction multiple-choice question task sourced from field notes increased from 33.7% to 87.1%, demonstrating the potential value of using richer biological metadata during training. However, generating open-ended biological descriptions remained more challenging than predicting predefined categories.
Tier 3: Scene understanding
The models also improved on tasks requiring an understanding of how vocalizations relate to one another within a recording. These included identifying which species vocalized first, which sounds had the highest or lowest frequency, and whether vocalizations overlapped. Synthetic audio training provided examples with known event timings and acoustic characteristics, helping models learn these relationships. However, fine-grained event counting remained difficult, and improvements were not consistent across every task.
Tier 4: In-context learning
One particularly promising result was the model’s improved ability to recognize unfamiliar species using audio examples provided at inference time. On the unseen-species matching task, where the model compares a recording of an unseen species against candidate species recordings, accuracy increased from 27% to 65%. This capability could be particularly useful for bioacoustic research, where labeled recordings are often scarce. Rather than training a new classifier for every species or call type, researchers could provide a model with audio examples and ask it to identify matching sounds in other recordings.
“Rather than training a new classifier for every species or call type, researchers could provide a model with audio examples and ask it to identify matching sounds in other recordings.”
Expanding What Bioacoustic Models Can Do
BEANS-Next provides a framework for evaluating the capabilities that bioacoustic research needs, while ROOTS offers new training resources to help develop them. Both are now openly available to support further research and development across the bioacoustics community.
This research has also helped inform the development of NatureLM-audio v2.0, the next generation of our large audio-language foundation model for bioacoustics. Building on the work presented here, we’re continuing to develop models that can support a wider range of bioacoustic research tasks, giving scientists more flexible tools to investigate animal sounds and communication.
We’ll be sharing more about NatureLM-audio v2.0 and its new capabilities soon. Stay tuned!