🔍 Read the full analysis: The Evidence Microsoft Cited For Its AI Beats Doctors Claim on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Microsoft’s MAI-DxO system scored 83.2% on 304 complex medical cases in a study published in Science, compared with 36.1% for generalist physicians using reference materials. The results came from a text-based simulation of historical cases, not care of real patients, and the comparison did not include specialists or physician teams.
Microsoft reported that its MAI-DxO diagnostic system identified the final diagnosis more often than generalist physicians on two sets of historical cases, in a paper published October 22 in Science. The study recorded 83.2% accuracy on 304 complex cases, versus 36.1% for physicians using reference materials, but it tested text-based simulations rather than diagnosis and treatment of live patients.
The paper, “Sequential diagnosis with language models,” describes a system that uses a language model as a panel of virtual physicians. The agents propose possible diagnoses, request additional information or tests, and challenge one another over several rounds. The study evaluated both the final diagnosis and estimated test costs. Microsoft’s reported results put MAI-DxO at 80.3% on 325 emergency department cases, compared with 46.8% for physicians.
The complex-case comparison used records from the New England Journal of Medicine. Physicians reached 36.1% accuracy with textbooks and internet access and 22.3% without those resources, according to the Science paper as described in the source report. Microsoft’s results also show that GPT-5 without the MAI-DxO orchestration system scored 78.9% on the complex cases. The orchestration layer added about four percentage points, while the paper estimated lower testing costs for the system than for GPT-5 alone.
The benchmark began with a short case summary. Participants had to request further details from a simulated gatekeeper holding the case file; each test carried an estimated price. The study therefore examined how a system gathers information as well as whether it reaches the published diagnosis. Some methodological details cited in the source report come from the earlier 2025 preprint, and may not be identical to the final paper.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the Test Design Matters
The findings matter because they put a prominent number behind Microsoft’s effort to position AI as a tool for medical diagnosis. A system that can ask useful questions and select tests could, in principle, help clinicians manage complex cases. But the study establishes performance on a particular benchmark, not that MAI-DxO improves patient outcomes or can safely replace clinical judgment.
The gap between the AI and doctors should be read alongside the comparison group and case selection. The physician participants were primary-care doctors and hospital generalists working individually. The cases were difficult teaching records that would often be referred to specialists or discussed by care teams. The test did not compare MAI-DxO with those specialists or teams, so it cannot show how the system performs against the people most likely to handle such cases in practice.
The benchmark also cannot establish how often an AI might raise an unnecessary alarm. The source report says the NEJM cases included no healthy patients or harmless explanations, making it impossible to measure false alarms in the study. That limitation matters for hospitals and patients: a tool can perform well at recognizing difficult diseases in a selected case set yet still need testing for how it behaves across routine care.
Products Worth Considering
Samsung 32-Inch Movingstyle Essential 4K UHD Smart Monitor with Adjustable Rolling Stand
Canon SELPHY CP1500 Wireless Compact Photo Printer Bundle with RP-108 Ink and Paper Set, White
Canon SELPHY CP1500 Wireless Dye-Sublimation Photo Printer Bundle with 108 Sheets of Paper and 3 Ink Cartridges
As an affiliate, we earn on qualifying purchases.
From Preprint to Science Paper
An earlier version of the research appeared as a non-peer-reviewed preprint in June 2025. That version reported a more dramatic comparison, with the AI scoring 80% and physicians 20% on the complex cases; the paper published in Science reports updated figures and includes an open-book physician group. The final paper was received in April and accepted in August, according to the source material. All 15 listed authors had Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman.
Microsoft’s Sequential Diagnosis Benchmark was designed to differ from a conventional medical exam, where all case information is presented at once. It instead makes the system decide what to ask or test for next. The source report says a model graded diagnoses against the published final answer and that clinician re-grading of a sample showed good, but not perfect, agreement. Test prices were estimates mapped to billing codes and a 2023 U.S. hospital price list, with a fixed physician-visit cost; the authors described these as first-order approximations.
Publication in a peer-reviewed journal means the research was reviewed before publication. It does not mean outside researchers have independently reproduced the results. The figures remain the study authors’ reported results, rather than independent confirmation of performance in clinical settings.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft CEO of AI Mustafa Suleyman, speaking to TechCrunch
What the Scores Cannot Establish
The study does not show whether MAI-DxO can diagnose patients accurately in a hospital or clinic, where information may be incomplete, symptoms change, and examination findings or imaging matter. It also does not establish whether using the system changes treatment decisions, reduces diagnostic delays, prevents harm, or improves patient outcomes. Those questions require evaluation during actual care.
It is also unclear how well the system would perform against specialists or teams, or on a broader mix of cases that includes common illnesses and patients without serious disease. The source report notes that some details about the benchmark and physician sample come from the preprint because the full Science article is behind a paywall; those details may differ from the final version. No independent replication of the findings is reported in the source material.
A Study With Patients Planned
The next development identified in the source material is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their usual care. That setting could provide evidence about how MAI-DxO works with real clinical information and workflows, although the source report does not provide a start date, study size, or outcome timeline.
Until those results are available, the Science paper supports a narrower conclusion: MAI-DxO performed strongly on the specific historical-case benchmarks reported by Microsoft. Whether it can assist clinicians safely and usefully in routine patient care remains untested in the evidence described here.
Key Questions
What did Microsoft’s study find?
Microsoft reported that MAI-DxO reached 83.2% accuracy on 304 complex historical cases, compared with 36.1% for generalist physicians using textbooks and internet access. On 325 emergency department cases, the reported scores were 80.3% for the system and 46.8% for physicians.
Did the study test the AI on patients?
No. The benchmark used historical, text-based case records in a simulation. The source material describes a separate prospective study planned at Beth Israel Deaconess Medical Center involving consenting patients alongside usual care.
Were the doctors compared with MAI-DxO specialists?
No. The comparison group consisted of primary-care doctors and hospital generalists working individually. The study did not include specialist physicians or clinical teams, according to the source report.
Does the study prove MAI-DxO is better than doctors?
It reports higher scores than the participating physicians on the selected benchmarks. It does not establish that the system outperforms specialists in clinical practice, improves patient outcomes, or is safe to use as a replacement for a clinician.
What happens next?
The source material points to a prospective study at Beth Israel Deaconess Medical Center, where consenting patients would interact with MAI-DxO alongside their normal care. Its timing, size, and results are not specified.
Source: ThorstenMeyerAI.com
