Mental health startup uses audio ‘biometrics’ to detect signs of depression and anxiety

Kintsugi’s AI software analyzes the human voice to detect depression and anxiety, providing clinicians with a “more comprehensive, 360-degree view of the patient” that is sometimes lost in virtual care.


“TIt’s a quick brown fox that jumps over the lazy dog, says Remma Silova Olsson, slowly and emphatically on Zoom.

She explains that a simple sentence holds enormous value for mental health care, and she smiles as she admits that it might be unclear how a silly phrase could be meaningful to a computer programmer and leader of an AI startup.

It contains every letter of the alphabet and sound in the English language, says Seiilova-Olson, an immigrant from Kazakhstan who is one of the founders and chief scholar of this short proverb. Kintsugi Wellness Vigilance. Kintsugi believes that these voices provide invaluable insight that can help mental health providers better support people with depression and anxiety.

The Bay Area-based company is building artificial intelligence software that analyzes short segments of speech to detect depression and anxiety. The so-called acoustic vital signs software is being integrated into clinical call centers, telehealth services, and remote monitoring applications to screen and triage patients who reach out for support, helping providers quickly and easily assess and respond to their needs.

“There isn’t a lot of clarity about major depression or anxiety.”

Kintsugi CEO and co-founder Grace Chang

Seiilova-Olson, 36, met co-founder and CEO Grace Chang, 40, a Taiwanese immigrant now residing in Berkeley, in 2019 at an open AI hackathon in San Francisco. Surprised to cross paths in a male-dominated event, the women began comparing notes about their own personal challenges in an effort to access mental health care: Seiilova-Olson struggled to secure a therapist during postpartum depression with her first child, and when Chang needed her own support, She said it took months for anyone from Kaiser to contact her again.

“Living in the Bay Area, you push a button and the car can come to you or the food comes to you,” Chang says. “But this was really a challenge.”

As engineers, they have looked at the dilemma differently than doctors might.

“We’ve seen this as an infrastructure issue, with a lot of people trying to get through that front door,” Zhang explains. “But there is not a lot of clarity regarding major depression or anxiety, low to moderate. And if we can bring this information to these frontline practitioners, we may have a chance to greatly alleviate that bottleneck.”

Kintsuji was born from this idea in 2019. It is located in a competitive space for emerging health technology companies such as The correctness of the ellipse And Winter Light Labs Acoustic biomarkers are used to detect mental health or cognition problems, based on research showing that certain language patterns and characteristics of a person’s voice can be associated with psychological or neurological conditions. Kintsugi last year raised $8 million in seed funding led by Acrew Capital, and in February it announced it had closed a $20 million Series A round led by Insight Partners, which valued the company at about $85 million, according to PitchBook.

Personal mental health facilities typically use questionnaires to measure the severity of patients’ anxiety or depression, scales known as PHQ-9 and GAD-7 scores. But during telehealth visits or phone consultations — where face-to-face interaction is lost, making symptoms difficult to recognize — Kintsugi technology helps bridge that gap.

Nisha Cumberbatch, Assistant Director of Public Health at Spora healthKintsugi, a provider focused on health equality and people of color, uses Kintsugi to assess women in a doula-led all-virtual maternal health program, spora mumas. The audio analysis tool, which Spora began using for patient consultations a few weeks ago, has helped Cumberbatch identify women who experience anxiety and depression before, during or after pregnancy, or may be at risk of developing it. When a patient starts talking to a Spora or doula doctor on Zoom, Kintsugi’s AI starts listening and analyzing her voice. After processing 20 seconds of speech, the AI ​​will spit out the patient’s PHQ-9 and GAD-7. The employee can then use the mental health score to determine which additional tests may be needed and the best way to advise or direct the patient to resources—such as a psychiatrist, cognitive behavioral therapist, or obstetrician.

Cumberbatch says Kintsugi’s technique allows her to “more vigilantly monitor her patients” and “then move forward with proactive recommendations about relieving their symptoms.” She adds that while it’s not intended to replace doctors or formal medical evaluations, it can be used as a screening tool “to allow us to have a more comprehensive, 360-degree view of a patient when we don’t have them in front of our face.”

“That technology… [allows] Let’s get a more comprehensive, 360-degree view of the patient when you’re not in front of us. ”

Nisha Cumberbatch, Assistant Director of Public Health at Spora Health

Dr. Jaskanwal Deep Singh Sara, a Mayo Clinic cardiologist who collaborated with Ellipsis and led research on the potential uses of acoustic biomarkers for heart disease, warns that while the technology is promising for healthcare, the field has a long way to go — making sure it is accurate, safe, and beneficial for patients. And doctors alike.

“It’s not ready for prime time by any stretch of the imagination yet,” says Dr. Sarah. Studies in psychiatry, neuroscience, cardiology, and other fields have shown an association between voice biomarkers and several conditions or diseases, but they haven’t shown how this relationship can be used to improve clinical outcomes, he says. Such research is “not like saying, ‘How can we exploit it in clinical practice, and how useful is it?” How effective is it in measuring an individual’s medical course? “If it does not provide any benefits in terms of how we manage it, the question is: Why would you do it?”

He says addressing these questions is “one of the many next steps we need to take on this” and that larger clinical trials are needed to answer them. “If it makes healthcare delivery cheaper or more efficient, or if it improves outcomes for patients, that’s great,” he adds. “But I think we need to prove that first with clinical trials, and that hasn’t been done.”

To address these issues and validate its software, Kintsugi Clinical studiesincluding the University of Arkansas for Medical Sciences, and the National Science Foundation has granted Kintsuji has multiple grants to intensify her research. The company is also seeking “de novo” clearance from the US Food and Drug Administration and continues to build its dataset to improve its machine learning models. (Data and ideas from Kintsugi voice journal appplus conversations with call centers or telehealth providers and clinical collaborations with multiple hospitals all become part of the massive data set that fuels Kintsugi’s AI.) Kintsugi distinguishes itself in the healthcare space with artificial intelligence – many technologies rely on external data from electronic health records.

The data set on individuals’ speech can be alarming—particularly in the field of mental health and wellness, which is widely considered a regulatory wild west. (These products and services are often not subject to the same laws and strict standards that govern how licensed physicians provide formal medical care to patients.) But Kintsugi’s founders say that patient privacy is protected because its technology isn’t what people say but how they say it matters. Patients are also required to sign their consent and care is not affected by their decision to opt in or out, according to the founders.

Kintsugi says it has served an estimated 34,000 patients. The company currently works with a large health system that includes 90 hospitals and clinics in 22 states, and is active in a care management call center that delivers approximately 20 million calls annually. It also partners with Pegasystems, which provides customer service tools for healthcare and other industries, to help charge payers and service providers handle incoming calls. Zhang says other clients include corporate payers in Fortune 10, pharmaceutical organizations and digital health apps focused on remote patient monitoring, but she hasn’t yet been able to share their names. Kintsugi’s clinical partners include Colorado Children’s Hospital, Joe Dimaggio Children’s Hospital in Florida, Chelsea Hospital, Westminster Hospital in London and SJD Hospital Barcelona for Children in Spain, Zhang said.

Prentice Tom, Kintsugi’s chief medical officer, adds that he is working with the University of Arkansas to explore how the tool can be used to identify patients who have suicidal thoughts, or an increased or severe suicide risk, as well as with Loma Linda University, to research how technology can be used to detect burnout among clinicians. The team is also looking at ways to expand availability and uses for younger and older patients, as well as for mothers and postpartums. Beyond the patients themselves, nurses are perhaps the biggest beneficiaries of Kintsugi’s work, according to the founding team: having a triage tool that helps reduce administrative work or time spent asking general questions enables nurses to move patients more seamlessly on their journey.

But Tom, a Harvard-trained emergency medicine physician and former faculty member in the Stanford Department of Emergency Medicine, says Kintsugi is now doing more than addressing infrastructure issues alone. It democratizes access to mental health care, Tom said, moving away from the doctor-centric model that caters to more people with depression significant enough to require medical evaluation.

“This tool actually creates a view of mental health in terms of mental health, where everyone has the opportunity to understand where they are on the spectrum, and that really divides treatment options far beyond the current infrastructure,” Tom said.

More from AI 50 2022

More from ForbesAI 50 2022: The Best AI Companies in North America Shaping the Future
More from ForbesThe AI ​​50 Methodology: How We Chose Our Honorees
More from Forbes$2 billion emoji: a hugging face that wants to be the launchpad for the machine learning revolution
More from ForbesInnovative AI Waabi Adds Self-Driving Veterans in Race to Market Robot Trucks