It’s a beautiful day and you are off on a walk in the mountains. You notice some flowers that are blooming by the side of the trail and you stop to study their exquisite structure and color. What’s the name of these plants? You photograph the plants and the flowers. As you walk away you wish that your botanist friend Agnes were with you, she would have told you the name instantly. You will try and remember to share the photos with her as soon as you get back to cell coverage.

Actually, you can do better than that. If you download the app iNaturalist to your phone it can instantly recognize most any plant or animal that you will encounter on your walks. The app is powered by computer vision algorithms that recognize more than 50,000 species of plants and animals from photographs. A community of naturalists is brought together by the app and reviews each observation, helping members of the public confirm their identifications. The algorithm continuously improves by learning from the experts, and the experts learn what they do not know from the algorithm. Scientists can benefit from the data that is thus collected, by you and hundreds of thousands of other citizen scientists, to estimate the range of species and to monitor the impact of climate change on the biosphere.

Project Visipedia, a collaboration between Professors Perona at Caltech and Serge Belongie at Cornell Tech, is at the foundation of iNaturalist and of its sister app Merlin Bird ID. Visipedia brings together communities of experts to share their knowledge and data. No expert knows it all, but the community is more than the sum of its parts. Powerful algorithms learn from experts to interpret the data and thus capture the knowledge of the community, so that it can be made available to anyone, anywhere at any time. Visipedia is about sharing knowledge and democratizing access to it.

FURTHER READINGS

Belongie, Serge, and Pietro Perona. "Visipedia circa 2015." Pattern Recognition Letters 72 (2016): 15-24. [PDF]

Branson, Steve, Grant Van Horn, and Pietro Perona. "Lean crowdsourcing: Combining humans and machines in an online system." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7474-7483. 2017.

Van Horn, Grant, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. "The inaturalist species classification and detection dataset." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8769-8778. 2018. [PDF]