Turning Motion into Meaning: the ESP journey to becoming Nat Geo Explorers
Katie Zacarian, CEO, Earth Species Project reflects on the ESP journey to becoming National Geographic Explorers
Two things happened this week that have me excited to share some news. The first is that dolphins are swimming in Biscayne Bay. I can see them playing as I write this, and I’ll add a grainy photo below. It is always thrilling to see them in this habitat.
The second is that Earth Species Project has been awarded a 2022 National Geographic Explorer Grant for our Self-Supervised Ethogram Discovery project, led by our Senior AI Research Scientists Benjamin Hoffman and Maddie Cusimano.
This award is a major milestone for ESP because it is the first institutional grant we’ve received recognizing our work. Personally, it is also a fitting milestone because this project—focused on developing machine learning models to interpret animal motion data—traces back to the very earliest days of ESP.
Professor Ari Friedlander is one of the world’s foremost experts on marine mammal behavioral ecology, and he introduces himself with the smile of someone who has been at sea in every kind of weather. When we met, it was a balmy day in Monterey Bay, California in July of 2019, and I had just hopped on to an expedition with him to collect data about humpback whales there.
As the head of the Lab for Biotelemetry and Behavioral Ecology at UC Santa Cruz, Ari and his colleagues have developed novel techniques to assess the health of whale species and the Monterey ecosystem using information collected from Customized Animal Tracking Solution (CATS) tags, which they were attaching to whales.
Simply put, the amount of information captured by these sensors—CATS and other animal-borne tags—is extraordinary. They provide insight into the orientation of the animal at each point in time, how it’s moving and changing position, while also capturing audio and video. Ari and his colleagues have innovated new ways for us to understand the experiences and health of cetacean species, and their discoveries have also given rise to unprecedented opportunities to interpret animal behavior.
It was clear from my first meeting with Ari that his research could change our comprehension of animal activity, and that the body of research that could follow from his work could fundamentally shift our understanding of all of earth’s species. Today, I am co-founder and CEO of the Earth Species Project, and proud to list Ari as one of our keystone research partners.
With the Friedlander Lab’s innovations in animal-borne tracking, ESP Senior AI Research Scientists Benjamin Hoffman and Maddie Cusimano were able to lead a project developing machine learning models to interpret meaning from motion data captured by sensors like CATS tags.
This project aims to provide researchers with new tools to interpret animal behavior recorded from tags. These tools include machine learning models which can predict an animal’s behavioral state based on its motion, and also include public benchmark datasets that researchers can use to evaluate the performance of behavioral models. Using these tools, researchers can get a better sense of how animals spend their time, as well as how external factors such as human disturbance may affect an animal’s behavior. For example, the dolphins out in front of my window here. We know their departure and return from the harbor can tell us more about the ecosystem and their needs, and moreover, what we might be able to do in order to mitigate against harms. It is clear to me - from biological research like Dr. Friedlander’s and machine learning research like Benjamin’s and Maddie’s - one day they could just tell us.
This important step of using the methods developed to understand how an animal's motion is related to its communication behavior gets us ever closer to ESP’s goal of decoding animal communication.
It is also an incredibly critical moment in time. We are seeing an extraordinary convergence of the powerful evolution of machine learning and large multimodal models alongside the growing urgency of the climate and biodiversity crises. ESP’s founding was based on an intuition that advances in natural language processing, developed with humans in mind, could be extended to the non-human domain through the development of latent representations of animal communication. And that the new understanding that would result had the potential to transform our relationship with the rest of nature.
This is the vision our Co-Founder Aza Raskin first impressed upon me, and a massive amount of technological advancement has happened since my eye-opening opportunity to “tag along” with Ari in 2019.
In the last year alone, we’ve witnessed models like GPT-3, AlphaFold, and DALL-E 2 transform what is possible in the fields of biology, healthcare, art, and design. Each new discovery in machine learning provides an opportunity to improve the lives of human beings and creates new possibilities to deepen our understanding of non-humans—chances to improve our relationships with them, and with each other.
With this generous grant from the National Geographic Society we are now able to develop a longer-view technical roadmap, one that harnesses the power of these advances in machine learning to ultimately decode communication of other species. We are incredibly honored to be recognized by National Geographic for this work.
Explorers Grants are awarded to individuals (in our case, our Co-Founder and President Aza Raskin) who “illuminate and protect our world through their work in science, exploration, education, and storytelling.” It is no surprise Professor Friedlander has also received an Explorer grant, and that Professor Christian Rutz of the University of St Andrews is also on the list.
Professor Rutz is a leading expert in wildlife tracking technologies and large-scale data projects, and like so many of our other collaborators, emblematic of the extraordinary team of research scientists and collaborators we have been able to amass over the past year, along with keystone partnerships with a wide array of researchers at leading institutions.
As CEO, it is simply my job to be the humble steward of all this incredible collaboration. And humbling is an understatement. All of the Nat Geo Explorers along with all of their researchers, scientists, collaborators, and donors, are pushing the boundaries of what is possible in service of others. They help us access and interpret new information, shining new light onto the unknown. At ESP, we are developing tools to help them do that—not just for ourselves but to give researchers worldwide opportunities to explore new frontiers of the unknown, expand their ability to listen, to understand, and ultimately to create a world in which all species can thrive.
