What happens when an AI chatbot takes an open-book exam in medicine?
A nationwide trial in Germany found that anchoring an AI chatbot to official clinical guidelines produced safe, helpful answers for rheumatology patients, making only one confirmed unsafe error across thousands of interactions.
Reading level
The full story with proper science words explained.
An open-book test
Picture taking a test where you cannot guess from memory. Instead, you can write an answer only if you find exact proof in an approved handbook on your desk. That simple rule is the core idea behind a rheumatology AI chatbot test run across Germany.
People with long-term joint conditions often spend months waiting between short doctor visits. Looking up facts online during painful flares can bring bad advice or total panic. To fix this gap, a team built ten condition-specific chatbots powered by retrieval-augmented generation (RAG). This method forces the software to search verified medical handbooks before it speaks. In this setup, the bot retrieves up to eight short passages using both keyword and math-based search. It then rewrites those passages into clear, plain language with sources attached.
High marks and few slip-ups
Over 1,534 user sessions, people asked 6,291 questions. The median conversation depth was three turns, which means most users had quick, focused chats. People rated 42.5% of the replies, giving 92.9% of those answers a thumbs-up. Among 602 surveyed users, 84.6% said the tool was easy to use, 84.1% found answers simple to understand, and 56.0% liked it better than an ordinary web search.
Checking thousands of answers by hand takes too long, so the researchers used an LLM-as-a-judge. This is a second software model trained by specialists to grade every reply for quality. It scored 95.3% of the answers as entirely safe and 79.1% as entirely correct, while marking 6.0% wrong. Across all 6,291 answers, only 11 were flagged as possibly harmful. When clinic doctors checked those 11 flags, they confirmed just one truly unsafe reply.
The trap of a printed book
The open-book setup was very safe, but it brought a surprising flaw: the tool became too cautious. The chatbot refused to answer 4.2% of queries, which was 263 questions in total. Yet human doctors reviewed those refusals and found that 17.1% were safe to address.
Furthermore, only 45.0% of all answers followed the handbooks completely. Science moves quickly, but the model relied on fixed documents. If health agencies approved a new drug after the text was written, that fact was missing from the shelf. When asked about missing facts, the bot either drew on general background knowledge or went quiet.
What we still do not know
This project was an observational study, which leaves open questions. It ran without user logins, so researchers could not verify real health records or count repeat visits by the same person. The system also had no live human safety monitor. We do not know whether using this tool improves daily health outcomes, helps patients take their medicine, or cuts down on clinic visits. For now, an open-book helper offers solid study support, but human doctors still hold the final say.
Science words
- Retrieval-augmented generation (RAG)
- A technique that forces an AI to search specific reference documents before drafting an answer, ensuring its outputs match verified texts.
- Conversation depth
- The total number of back-and-forth question-and-answer pairs exchanged within a single session.
- LLM-as-a-judge
- Using a separate large language model to systematically rate the quality, safety, and categories of text outputs at scale.
Check it yourself
This story is based on a real research paper in Journal of Medical Systems by Wilhelmi, Bartsch, Platt et al.. We write with AI help and check it against the paper, but the original is the final word.