Google's AI chatbot took medical histories from 98 real patients with no safety stops in a first clinic study

A patient with a new complaint books a prompt appointment with a primary care doctor. Before the visit, a chatbot asks what is wrong, follows up on the answers and writes a summary for the doctor to read. That arrangement has been tested many times with actors and written cases. A study published in The Lancet on October 8 tried it with real patients at a Boston hospital.

The chatbot, built by Google and called AMIE, got through 98 interviews without a supervising physician having to halt one on safety grounds. Patients mostly liked it, and the doctors who read its summaries mostly found them useful. The study was small, had no comparison group, was paid for by Google’s parent company and kept a physician watching every conversation. It shows that the idea can be tried safely under supervision. It does not show that patients were better off.

What the AI chatbot did before the clinic visit

AMIE stands for Articulate Medical Intelligence Explorer. It is built on a large language model (LLM), the kind of artificial intelligence (AI) software behind consumer chatbots, and tuned for the back-and-forth of a medical interview.

The study was run with a teaching hospital in Boston. Its registration names the lead sponsor as Beth Israel Deaconess Medical Center, with Google as a collaborator. A lead sponsor in a registry is the body responsible for running a study, which is separate from who pays for it.

The design was simple. In the paper’s description, English-speaking patients aged at least 18 years interacted with the Articulate Medical Intelligence Explorer (AMIE) up to 5 days before a single-complaint urgent primary care appointment. Urgent here means a prompt visit for a new problem, not an emergency. Google describes the patients as booked for new, non-emergency, episodic complaints, either in person or via a telehealth platform.

Some kinds of visit were left out. The registration lists exclusions, among them cases in which the visit reason is related to pregnancy or is a psychiatric concern.

Patients typed their answers in a text chat, and they were not alone with the software. These AI-driven text-chats were overseen by a physician using a live video-call with screen-sharing, by Google’s account.

When the chat ended, the supervisor went over it with the patient. The plan filed before the study began was that the safety supervisor will discuss with the patient and correct any errors, hallucinations, or problems with the AI. A hallucination, in this field, is a statement the software makes up.

The doctor then received the paperwork. AMIE transcripts and summaries were shared with PCPs before the visit, PCPs being primary care providers. The appointment itself went ahead as usual.

The numbers were modest. From April to November, 2025, 114 patients were enrolled with 98 completing both the AMIE interaction and the PCP appointment.

How safe the chatbot’s conversations were

Safety was the first thing measured, and it was measured in a specific way. Physician safety supervisors monitored all interactions and were trained to intervene on the basis of predefined safety criteria.

Google lists four. A supervisor was to stop the chat over an immediate worry about harm to the patient or to others, over marked distress caused by the AI, over possible clinical harm identified by the supervisor based on the conversation, or if the patient asked to end the session.

None of those happened. Zero conversation safety stops were required on the basis of prespecified criteria.

That is not the same as zero problems. Safety supervisors noted one hallucination and added clinical information in five interactions. So in as many as six of the 98 completed chats, a physician corrected the record or filled a gap.

The distinction matters for reading the headline number. No safety stops means no supervisor judged that a conversation had to be halted. It does not mean that every statement the chatbot made was right, and it says nothing about what would have happened had nobody been watching.

How the chatbot’s diagnoses compared with the doctors’

The interview was the thing under test, but the researchers also had AMIE draw up a list of possible diagnoses and a treatment plan after each chat. These were kept away from the visit. The registration says they will not be shared with the patient or physician, but will be generated after the fact and compared with the actual diagnosis and management plan.

A separate panel then scored both sets. According to Google, a panel of clinical evaluators who were not involved with urgent care consultations rated differential diagnoses and management plans from AMIE and PCPs in a blinded and randomized manner. A differential diagnosis is the list of conditions that could explain a patient’s symptoms.

On the quality of those lists, and on whether the plans were appropriate and safe, the evaluators found no clear difference between the chatbot and the doctors. On two practical measures the doctors did better. In Google’s summary, PCPs outperformed AMIE in the practicality and cost effectiveness of their plans.

Google’s authors offer a reason. The chatbot could not see the patient’s medical record and, in their words, did not have the ability to perform a physical exam. The doctors had both, and Google suggests they could also draw on experience of their own clinic.

How often was the chatbot’s list right? Each patient’s record was reviewed weeks later to settle on a final diagnosis. Google reports that AMIE’s list included that diagnosis in 90% of cases, and that the system accurately identified the final diagnosis as its single most likely diagnosis in 56% of all cases evaluated.

What patients and doctors made of the AI interview

Clinical evaluators also graded the conversations themselves, and rated them highly. AMIE’s conversations were rated well in 87-100% of cases (17 criteria) by clinical evaluators. Patients were less uniform. Asked about 16 criteria, the share giving a favorable rating ranged from 48% on the weakest to 96% on the strongest. On at least one count, then, about half of patients were not won over.

Attitudes shifted all the same. Patients filled in a standard questionnaire on their views of AI before the chat, after it, and again after seeing their doctor. Their views of AI improved after interacting with AMIE and stayed higher after the appointment.

Doctors were asked too, though not all of them answered. PCPs completed post-surveys in 60 of 98 cases, including 44 cases in which they reviewed the AMIE transcript before the visit. In those 44, the doctors said the transcript helped them prepare in 33, or three-quarters. In 25 of the 44 they reported that it might have changed what they did.

That last figure cuts two ways. It suggests the summaries carried information the doctor would otherwise have lacked. It also means that a chatbot’s account of a patient may have shaped decisions in more than half of the visits where it was read, which is why its accuracy matters.

The denominators deserve a note. Of 98 completed visits, the survey shows the doctor reading the transcript beforehand in 44. The study’s picture of how doctors used the tool rests on fewer than half of the visits.

How the study fits with earlier tests of medical AI

Until now AMIE’s record came from simulations. In a study published in Nature last year, Google researchers compared it with primary care physicians in text consultations with trained actors playing patients. The study included 159 case scenarios from providers in Canada, the United Kingdom and India, 20 primary care physicians compared to AMIE, and evaluations by specialist physicians and patient-actors. The chatbot came out ahead. AMIE demonstrated greater diagnostic accuracy and superior performance on 30 out of 32 axes according to the specialist physicians.

Results like that raised the question the new study begins to answer: does performance with actors carry over to real patients? Other researchers had found reasons for doubt.

A team at the University of Oxford tested what happens when ordinary people use language models to work out what might be wrong with them. It ran ten medical scenarios in a controlled study with 1,298 participants. The models did well when left to themselves. Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9% of cases. People using them did far worse. In the study, participants using the same LLMs identified relevant conditions in fewer than 34.5% of cases, no better than people who used whatever source they liked.

The Oxford authors drew a pointed conclusion: “Standard benchmarks for medical knowledge and simulated patient interactions do not predict the failures we find with human participants.”

Doctors have not always fared better with these tools than the public. A randomized trial led from Stanford University gave physicians a set of written cases, with or without a chatbot to help. Fifty physicians took part. The median diagnostic reasoning score per case was 76% with the chatbot and 74% without, a gap well within the range of chance. The software working by itself beat both groups. The LLM alone scored 16 percentage points higher than the physicians using conventional resources. One of that trial’s authors, Adam Rodman, is also the last-named author of the new study.

Taken together, the three studies describe a gap. Language models can outscore doctors on paper. Getting that performance to survive contact with a real person typing real symptoms is a separate problem. The new study is a first attempt at it, with a system built for interviewing and with a physician standing by.

What the clinic study cannot show

Whether care improved. There was no comparison group. Google’s own authors write that the study “did not include controlled comparisons and therefore does not support quantitative claims on the efficacy of this intervention compared to a baseline workflow”.

What happens without a supervisor. A physician watched every chat live and went over it with every patient. That is far more oversight than routine use would allow. The absence of safety stops was observed under those conditions.

Who took part. One hospital, English speakers and 114 volunteers. Google notes that the group was younger than usual for the setting: by its count, more than half of the urgent care visits at the clinic during the study period were by people over 60. Visits about pregnancy or mental health were excluded. People who agree to chat with an AI before seeing a doctor may also be more at ease with the technology than those who decline.

Typing only. The interface was text. Google lists that first among its limitations: the study’s text-only chat interface does not fully capture the rich, multimodal nature of clinical care. Multimodal means drawing on voice, appearance and examination as well as words.

The diagnostic comparison. The chatbot’s diagnoses were produced after the fact and never used. Showing that its lists looked as good as the doctors’ to a panel is different from showing that patients would have been diagnosed correctly had anyone acted on them.

Small numbers. Ninety-eight interviews cannot say much about rare, serious errors. A mistake that happens once in 500 conversations would most likely not appear at all.

Who paid. The study was funded by Alphabet, the parent company of Google, and Google researchers are among its authors.

The authors do not claim more than that. Their conclusion is that the study shows the initial feasibility of conversational AI in a real-world setting, and Google says larger studies with comparison groups are planned.

What a new symptom means for a particular person depends on their history and an examination, which is assessed by their doctor.

In its first test with real patients, Google’s medical chatbot interviewed 98 people ahead of urgent primary care visits without a supervising physician having to stop a conversation, in a company-funded study at one hospital that had no comparison group and kept a doctor watching every exchange.