The Growing Challenge of Biodiversity Data
Citizen science projects are generating more data than ever before. Global biodiversity databases now contain millions of species observations, creating an incredible resource for ecological research.
While collecting the data is not a problem, researchers still need to analyze the sounds hidden within thousands or even millions of recordings. Many contain background noise. Others capture multiple species calling at the same time. Still others need a trained ear to hear the difference between two similar-sound species. Sorting through all of that information manually can take a huge amount of time.
This is a challenge Julia Tan is trying to tackle.
In a previous collaboration with ESP, Julia evaluated our flagship model, NatureLM-audio, for use in processing and analyzing hundreds of thousands of frog recordings, and found strong performance and potential time-savings, even without fine-tuning.
In a paper presented at the AI for Non-Human Communication workshop at NeurIPS 2025, Julia shows how an AI pipeline can help researchers extract useful information from noisy citizen science recordings, supporting biodiversity monitoring at scale.
What motivated you to develop this pipeline for FrogID, and what problem were you trying to solve?
The core motivation behind developing this pipeline for FrogID was the growing bottleneck created by manual validation. Citizen science initiatives like FrogID produce a large volume of valuable acoustic data, but expert review remains time-intensive, which limits the speed and effectiveness of species monitoring and delays timely management action. At the same time, while there is now more ecological data available than ever before, much of it is noisy, weakly labelled, or only coarsely annotated, making it difficult to use in conventional machine learning pipelines.
We were also motivated by the rapid progress taking place in other areas of AI, particularly in fields such as human speech and bird call recognition. However, many other taxa, including frogs, have not been represented to the same extent in these advances. This raised an important question for us: how can we take the scale of citizen science data and combine it with progress in adjacent AI domains to support a more specialised ecological application? More broadly, we wanted to help bridge the gap between carefully curated benchmarks and the far messier conditions of real-world monitoring data. The aim was not simply to automate parts of the process, but to do so in a way that maintained scientific value while reducing the burden on experts, improving the practicality of large-scale acoustic monitoring.
Why are citizen science audio datasets both so valuable and so challenging to work with?
Citizen science audio datasets are incredibly valuable because they allow data to be collected at a scale that would be difficult, costly, or even impossible to achieve through conventional field surveys alone. Programs like FrogID can gather recordings across a wide geographic area in a relatively short period of time, creating a rich source of ecological information. They can also provide access to places researchers may not otherwise reach, including private land, which makes up the majority of Australia.
“Citizen science audio datasets are incredibly valuable because they allow data to be collected at a scale that would be difficult, costly, or even impossible to achieve through conventional field surveys alone.
Just as importantly, citizen science encourages people to actively engage with their local environment, helping to build awareness of biodiversity while contributing meaningful observations for research and conservation. At the same time, these datasets can be challenging to work with because they are collected under highly variable real-world conditions. Unlike curated research datasets, there is no standardised recording quality, so recordings may include heavy background noise, overlapping sounds, or audio where the frog is not the clear focal species. Contributors also often record a bird or insect while believing they are hearing a frog. In addition, observations are often unevenly distributed, reflecting where people live and where they choose to record, which can skew datasets heavily toward common species and more populated areas.
involving a hybrid neural network and k-NN approach.
How does the pipeline help distinguish target animal sounds from background noise or other species?
The pipeline helps distinguish target animal sounds by first focusing on where the target signal is in a recording, before worrying about exactly which species it is. This is especially important in citizen science audio, where recordings often contain heavy background noise, overlapping calls, and multiple species at once. In these conditions, the first challenge is not fine-grained species identification, but isolating the part of the recording that actually contains the target animal. To do that, we used unsupervised source separation rather than traditional background noise removal. This was important because the target sound is not always the dominant or focal sound, so a standard denoising approach could accidentally remove or weaken the very signal we are trying to detect. After separation, each audio stream was analysed using windowed taxonomic classification with the NatureLM-audio foundation model. Because the recording already carried a clip-level label, we knew the target animal was present somewhere in the clip, just not exactly where.
NatureLM-audio made this localisation step possible by turning a difficult species-level problem into an easier decision across broader taxonomic groups, supported by its diverse pretraining across taxa. We also found that asking the model to choose between broader categories (i.e. frog vs bird vs insect) performed better than framing the task as a simple target-versus-non-target decision. The selected target-positive windows then acted as reliable proxies for time-localised labels, giving a bounding-box-like indication of where the sound occurred before being passed into an embedding model for downstream species-level classification.
How did collaboration between ecologists and AI researchers shape the design of this work?
Collaboration between ecologists and AI researchers was central to the way this work was designed. From the input side, expert validation played a critical role in ensuring label accuracy, particularly for the more difficult or ambiguous recordings. Rather than treating the model as a replacement for ecological expertise, the work was shaped around supporting, extending, and helping prioritise that expertise toward the recordings where it was most needed. This collaboration also influenced how predictions were presented.
“Rather than treating the model as a replacement for ecological expertise, the work was shaped around supporting, extending, and helping prioritise that expertise toward the recordings where it was most needed.
In ecology, scientific accuracy and transparency are especially important, so it is not enough for the system to simply output immediate verdicts. By providing interpretable uncertainty scoring alongside predicted classes, we ensure that ecologists can meaningfully incorporate the predictions into their workflows, instead of being forced to either accept or reject them outright. Ecological knowledge also shaped how the model outputs were visualised and interpreted. For example, understanding recent taxonomic changes was important when examining the embedding space, where some acoustically similar species clustered closely together. In some cases, these were species that had only relatively recently been considered distinct. In turn, the visualisation became a useful meeting point between ecology and AI, revealing patterns that reflected known ecological context while also suggesting how these tools could support future taxonomic and ecological insight.
How could this pipeline be adapted for other citizen science projects beyond frogs?
The key strength of this pipeline lies in the dataset annotation stage, which was designed from the outset to be flexible around the target animal rather than tied specifically to frogs. From an ablation study, we found that the annotation stage is the main driver of robustness under noisy, non-ideal conditions, which are typical of citizen science recordings. Adapting it mainly involves changing the target of the windowed taxonomic prediction step, while keeping the broader workflow intact. In particular, the sound source separation step is already unsupervised, so it does not depend on any one species group, and NatureLM-audio has been trained across a diverse range of taxa, allowing it to support broad taxonomic classification beyond frogs.
We also found that the pre-trained Perch embedding model could represent frog calls effectively, even though frogs made up only a small part of its original pretraining data, which also included insects, mammals, and other non-bird species. Together, these cross-domain models do most of the heavy lifting, allowing the final classification stage to remain relatively simple and just learn the class labels present in the new dataset. The pipeline could also support training multiple downstream models from the same raw recordings when more than one taxonomic group is present, further unlocking the potential of citizen science projects.
What do you see as the biggest opportunity for AI to support citizen science-driven conservation?
I think the biggest opportunity for AI in citizen science-driven conservation is its ability to turn large volumes of messy, real-world data into something that can be acted on quickly and reliably. Citizen science projects can generate huge numbers of recordings, images, and observations, but their conservation value often depends on how efficiently that information can be processed, prioritised, and fed back into informed decision-making. AI offers a way to do that at scale, helping to sort through data collected under non-ideal conditions and direct expert attention to the records that matter most.
That same opportunity extends to the people contributing the data. If AI can provide immediate, meaningful feedback, it can encourage participants to continue engaging and recording, especially when something unusual or important may be present. In that sense, AI can assist in strengthening the loop between participation, validation, and conservation action. Used well, it has the potential to make citizen science projects more scalable, more responsive, and ultimately more impactful for conservation.
Julia Tan is a Machine Learning Engineer from Sydney, passionate about applying AI to biodiversity conservation. She has a background in Mechatronics Engineering from the University of Sydney and completed her Honours research at NASA’s Jet Propulsion Laboratory, where she worked on a self-supervised learning approach to robotic perception for an autonomous lunar rover designed to explore the far side of the moon. Before joining the Australian Museum Herpetology team, she worked on AI systems across wearable neurotechnology, distributed computer vision, and consumer robotics.
More recently, she has been working on the FrogID project, applying machine learning methods to frog call recordings to support automatic species identification. Outside of work, she enjoys hiking and birdwatching, as well as making electronic music and DJing.