Inside Google DeepMind Vision for AI in Medicine with Med-Gemini

May 13 2025 18:42
In a recent interview with Professor Hannah Fry, Joelle Barral outlined a vision where AI doesn't replace doctors but instead augments them, handles tedious documentation, and makes advanced healthcare more accessible worldwide.


"I'm hoping AI contributes to bringing back the joy of practicing medicine," says Barral, pointing to how technology is helping pay back what she calls the "digital debt" created when physicians were first required to enter enormous amounts of data into electronic health record systems.


Google DeepMind's journey in healthcare AI began with what Barral describes as "narrow AI" - focused tools designed to perform specific tasks. One of their early successes came in diagnosing diabetic retinopathy, the leading cause of preventable blindness worldwide. Their AI system became adept at analyzing fundus camera images of the retina, grading them on a scale from 1 to 5 to indicate disease severity.


What's particularly fascinating about this project was the discovery that even expert ophthalmologists frequently disagreed on diagnoses. To establish reliable "ground truth" for training their models, the team needed input from 50 different ophthalmologists. In some cases, the same image received wildly different ratings from different doctors - an insight that helped shape how the AI was developed.


Today, Google has advanced well beyond these narrow applications with Med-Gemini, a specialized version of their Gemini AI model fine-tuned specifically for healthcare. Med-Gemini inherits Gemini's advanced reasoning capabilities and multimodal understanding (the ability to process text, images, video and more) while being trained on de-identified medical data.



The capabilities of these systems now extend far beyond image interpretation. Barral described a tuberculosis detection system that can analyze audio recordings of coughs - something that sounds like science fiction but has proven effective in testing. "You record a cough, and it can tell if you've got TB," Professor Fry asked incredulously.


Perhaps more surprising is how these AI systems sometimes see patterns invisible to human experts. Researchers discovered that AI could determine a patient's biological sex by analyzing the blood vessels at the back of the eye - something no ophthalmologist could reliably do. This opens possibilities for detecting conditions that might present differently based on sex, even when that wasn't what the system was initially designed to identify.



While these specific applications are impressive, Barral believes AI's greatest potential lies in its ability to integrate information across previously isolated domains. For instance, Google DeepMind is collaborating with France's Curie Institute on women's cancers, using AI to analyze everything from cellular-level data to genomics and imaging.

"We're really hoping that by combining those modalities, by getting to a deeper understanding at the cellular level of what's going on, we'll finally be able to crack, for example, triple-negative breast cancers or some of the more advanced cancers that, today, we don't have good solutions for," Barral explained.

This holistic perspective extends to lifestyle factors as well. Sleep patterns, step counts, social interactions, and eating habits all influence health outcomes. AI systems can potentially integrate these diverse data streams to provide insights that would be nearly impossible to derive manually.



The concept of "digital twins" - virtual replicas of patients or specific organs - represents another frontier in medical AI research. These simulations could allow researchers to test treatment approaches without putting real patients at risk.

"For clinical trials, that would be exactly that," Barral confirmed when asked about creating simulated humans or organs to reduce the number of actual patients needed for research.

This approach could be particularly valuable for understanding rare diseases or testing treatments for conditions where enrolling large numbers of patients proves challenging. However, it also raises important questions about data privacy and patient consent.


Perhaps most intriguing is Barral's vision of future AI as being "like the physician in your family" - a trusted advisor who knows your history, understands your tendencies, and can help navigate the healthcare system.

"Not all families have physicians," she noted. "I'd like to think that the AI we'll build tomorrow will provide everyone with the equivalent of a physician in your family."


This vision is being pursued through research projects like AMIE (Articulate Medical Intelligence Explorer), which explores how conversational AI might handle doctor-patient interactions. AMIE not only demonstrates strong diagnostic capabilities but also shows promise in clinical management reasoning and longitudinal care - understanding how diseases develop over time and how patients respond to treatments.



When questioned about the things AI might never be able to do in healthcare, Barral had a surprising response about empathy - often considered uniquely human.

"We always check that our models have good bedside manners. And they actually do. They're not bad at all," she said. "They can adapt to their audience in a way that few human beings can. They can leverage the right language for a five-year-old, or a 70-year-old, or a 40-year-old."

The latest versions of Med-Gemini have demonstrated remarkable capabilities in this regard, able to communicate complex medical information in ways tailored to individual patients' needs, language abilities, and comfort levels.


Despite these advances, Barral emphasizes that AI remains an assistant to human doctors, not their replacement. When asked what happens if an AI diagnosis disagrees with a doctor's assessment, her answer was clear: "At the end of the day, the doctor is really the one making the call."

This approach addresses concerns about over-reliance on technology or the potential erosion of medical skills. The goal instead is to handle routine tasks, provide decision support, and enable physicians to focus more on patient care and complex cases.

The comparison Barral makes is to spell-check: "Sometimes I disagree with my spell checker, with accents on words in French or things. It's not perfect. But I will double-check."



The potential impact of these technologies extends far beyond well-resourced healthcare systems. Google's diabetic retinopathy screening program has already screened 700,000 people in Thailand, with plans to expand tenfold in coming years.

This global perspective is central to Google DeepMind's vision - using AI to expand access to quality healthcare worldwide, especially in regions facing physician shortages or limited medical infrastructure.

Med-Gemini represents the cutting edge of medical AI, demonstrating unprecedented capabilities across various healthcare domains. Recent research papers show the system achieving 91.1% accuracy on the MedQA benchmark and making significant advances in interpreting complex 3D scans, genomic information, and generating radiology reports.

But perhaps the most revolutionary aspect is AMIE's ability to reason about disease progression over time, therapeutic responses, and medication management - moving beyond single-point diagnosis to longitudinal patient care.

While Barral is cautious about predicting overnight transformations in healthcare delivery, she sees enormous potential in reducing the role of serendipity in medical care. As she puts it:

Research Paper: Capabilities of Gemini Models in Medicine
Research Paper: Advancing Multimodal Medical Capabilities of Gemini
Research Paper: Towards Conversational AI for Disease Management