Researchers Release MedRealMM, First Real-World Benchmark for AI Medical Consultants Using Actual Patient Data

A new dataset called MedRealMM, built from 5,620 de-identified patient-doctor interactions across 64 clinical departments in China, benchmarks large language models on multimodal medical consultation tasks. The evaluation of 19 LLMs reveals that while some frontier models match physician performance on positive clinical criteria, they fail significantly on safety and error avoidance—a critical gap for clinical deployment.

Why it matters: As AI enters clinical practice, realistic benchmarks grounded in actual patient interactions are essential for understanding whether LLMs can safely assist physicians; this dataset addresses the industry's reliance on synthetic or incomplete evaluation frameworks.