Listen Learner: Automatic Class Discovery & One-Shot Interactions for Acoustic Activity Recognition

Acoustic activity recognition has emerged as a foundational element for imbuing devices with context-driven capabilities, enabling richer, more assistive, and more accommodating computational experiences. Traditional approaches rely either on custom models trained in situ, or general models pre-trained on preexisting data, with each approach having accuracy and user burden implications. We present Listen Learner, a technique for activity recognition that gradually learns events specific to a deployed environment while minimizing user burden. Specifically, we built an end-to-end system for self-supervised learning of events labelled through one-shot interaction. Our results show that our system can accurately and automatically learn acoustic events across environments (e.g., 97% precision, 87% recall), while adhering to users’ preferences for non-intrusive interactive behavior.
Paper Citation:
Wu, J., Harrison, C., Bigham, J. and Laput, G. 2020. Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition. In Proceedings of the 38th Annual SIGCHI Conference on Human Factors in Computing Systems. CHI '20. ACM, New York, NY.

[@DanielSMatthews]

Very impressive work, now try testing it in a household that has two or more children and a dog. ;-)

[@giancarlorsilva]

What an awesome project!
I'm thinking in some possibilities here. When system is ready and receive continuous updates, the blind, speech and/or hearing-impaired people will take a lot of advantage of this resource.
Will help them to be more secure and stay tuned to something that they want.
I'm very proud of you!

[@TonyDadda]

100% genius. Thanks for sharing.

[@kechto]

This was very interesting!

[@ryanh8764]

Very cool thanks

[@emanuelperez1898]

This is so cool! Will this be available for consumer use in the future?

[@herp_derpingson]

What if I live in a city in an apartment adjacent to a busy road?
Will it keep asking me whether the sound it is hearing is a Toyota or a Land Rover?