Smart Devices Stop Eavesdropping With Kirigami

In an age where smart speakers and devices listen constantly to our environments, a new technology promises to maintain the benefits of audio sensing while preventing your private conversations from leaving your home. Carnegie Mellon University researchers have developed Kirigami, an innovative on-device filter that detects and removes human speech before audio data is processed for activity recognition.

This breakthrough addresses a growing privacy concern: while sound data can power valuable applications like health monitoring and environmental sensing, it can also capture sensitive personal information from our conversations.

“The data contained in sound can help power valuable applications like activity recognition, health monitoring and even environmental sensing. That data, however, can also be used to invade people’s privacy,” explained Sudershan Boovaraghavan, who earned his Ph.D. from Carnegie Mellon’s Software and Societal Systems Department.

Unlike previous privacy solutions that merely altered audio data, Kirigami takes a more fundamental approach by filtering speech at the source. This distinction has become crucial with the rise of sophisticated AI speech recognition systems like OpenAI’s Whisper, which can reconstruct conversations from processed audio that was once considered secure.

“Given the sheer amount of data these models have, some of the prior techniques would leave enough residual information, little snippets, that may help recover part of speech content,” said Yuvraj Agarwal, an associate professor at Carnegie Mellon. “Kirigami can stop these models from having access to those snippets.”

The technology works as a binary classifier that determines whether speech exists in audio recordings. What makes Kirigami particularly valuable is its lightweight design, allowing it to run on even small, affordable microcontrollers. By processing data “on the edge” – directly on the device rather than in the cloud – sensitive speech never leaves the microphone.

Kirigami also offers customizable privacy settings. Users can set an aggressive threshold that prioritizes removing all speech but might also eliminate some non-speech sounds, or a less aggressive setting that preserves more environmental audio while accepting a slightly higher risk of speech content passing through.

“Kirigami cuts out most of the speech content but not the other ambient sounds that you care about for activity recognition,” said Haozhe Zhou, a doctoral student who co-led the project. “You can still couple it with prior techniques to give you additional privacy.”

This balance between privacy and utility is critical for the many beneficial applications of audio sensing. For instance, Mayank Goel, an associate professor at Carnegie Mellon, uses audio sensing to “remind people living with dementia of daily tasks, monitor children with attention-deficit/hyperactivity disorder for behavioral abnormalities, and assess students for signs of depression.”

“These are just examples that are being done in our labs,” Goel noted. “You will find similar scenarios all across the world where you need noninvasive data from the person about their daily life.”

The research demonstrates a new path forward for smart device manufacturers and developers. Rather than forcing consumers to choose between beneficial technology and personal privacy, Kirigami suggests we might have both. As smart home technology becomes increasingly ubiquitous, solutions like this will be essential for maintaining user trust and protecting sensitive information.

The research team believes developers could easily adapt Kirigami to meet various privacy requirements across different applications. Their work appears in both the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies and ACM MobiCom ’24.

As our homes fill with more listening devices, technologies like Kirigami offer hope that we can benefit from ambient intelligence without sacrificing our most private conversations.