How Abridge Uses AI to Help Doctors Spend More Time With Patients, with Zachary Lipton, Chief Technology & Science Officer at Abridge and Raj Reddy Associate Professor of Machine Learning at Carnegie Mellon University

The U.S.’s medical system is broken in many ways, but here’s one many don’t know: for every hour that a doctor spends with a patient, they spend another 2 hours doing paperwork. Abridge is using AI to help fix that by transcribing and networking medical conversations so doctors can spend more time doing what matters: helping patients.

Speech to text in medicine is incredibly difficult: it requires intense specificity, and foundation models aren’t trained on medical corpus words (e.g. misreading Ozempic as Olympic). I sat down with Abridge’s CTO/CSO to talk all about how they built their AI, from creating systems for introducing new vocabulary to a UI that encourages quick flywheel feedback.

On where foundation models fall short for vertical specific applications like medicine:

“Someone might think they can grab off the shelf automatic speech recognition models like Whisper with no further modifications or training. We’ve made incredible progress in speech, but it struggles with medical speech in particular. It transcribes Ozempic as Olympic. There are 500 different ways to misspell Mounjaro. A lot of the medical lexicon are new and unusual words that these models just aren’t trained on.”

On the difficulty of doing evals with speech:

“Whenever you’re generating free text, you always run into this problem. When you talk about something like classifiers, it’s clear which is the right and wrong class. There’s very high agreement, and you can automatically tabulate model performance. When it comes to machine translation, there might be multiple correct translations, and the golden one sitting in the dataset isn’t the one that the model output, but the model output isn’t wrong.”

On how AI product development is a new thing:

“AI product development is a new thing. There’s a very mature software development playbook, even though not everyone does it the same. How PMs, design, engineering, and infrastructure coalesce to build a software product. This other side of AI is different: interaction design decisions influence what data you get, which influences what you do on the modeling side, which influences what you build. Sometimes ideas come from ML, sometimes from customers, sometimes from product intel…this new world is what’s the most exciting thing in AI for me.

‍