Microsoft recently launched a new AI system called MAI-DxO, aiming to significantly improve the diagnostic accuracy of complex medical cases. According to reports, the system's diagnostic accuracy is four times that of experienced doctors and can reduce medical costs by nearly 70%. This technology was evaluated through a new benchmark test that simulated a real, step-by-step diagnostic process.
The research team detailed the system in their paper "Sequential Diagnosis Using Language Models," stating that it significantly outperforms human doctors in terms of accuracy and cost-effectiveness when dealing with challenging cases. To provide more realistic test results, the team created the Sequential Diagnostic Benchmark (SDBench). Unlike traditional medical AI tests, SDBench does not provide all information at once but simulates the sequential process of clinical decision-making.
SDBench's test cases come from 304 complex cases in The New England Journal of Medicine. In the test, doctors or AI diagnosticians first receive a brief case summary and then actively request more information by asking targeted questions or requesting tests to obtain detailed data. The system uses a "gated model" to control the release of information and can even generate synthetic results for certain tests to prevent accidental leakage of hints.
In the test, 21 experienced doctors from the United States and the United Kingdom had a diagnostic accuracy rate of 19.9%, with an average cost of $2,963 per case. In contrast, Microsoft's MAI-DxO combined with OpenAI's o3 model achieved an accuracy rate of 79.9% and an average cost of $2,397. The main advancement of MAI-DxO lies in the significant reduction in costs. Although the o3 model achieved the highest accuracy of 78.6% in standard models, its average cost was as high as $7,850. However, diagnoses using MAI-DxO saw a slight increase in accuracy while costs were reduced by nearly 70%.
MAI-DxO enhances performance by simulating a virtual doctor team. The team consists of "hypothesis doctors," "test selection doctors," "questioning doctors," "cost monitoring doctors," and "checklist doctors." This structure aims to prevent the system from prematurely fixing on a single hypothesis.
However, researchers also acknowledge some limitations. SDBench is based only on complex teaching cases and does not reflect the distribution of common diseases encountered in daily medical practice. Cost calculations are rough estimates and do not consider various factors in the real world. Additionally, the participating doctors were general practitioners who typically refer complex cases to specialists, and their assessments did not use external resources.
Key points:
🔍 The MAI-DxO system has four times the diagnostic accuracy of experienced doctors and reduces costs by nearly 70%.
💡 The SDBench test simulates a real, step-by-step diagnostic process, improving accuracy by dynamically requesting information.
🩺 The study indicates that the system performs well, but there are limitations and challenges related to complex cases.