SynthesisJMIR medical informatics2025
Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis.
Synthesis in JMIR medical informatics, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT07632859 (Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes), which is not on this map. Cited by 36 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes: A Retrospective Study of 600 Cases
Who cites it
36 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Application of Large Language Models in Chronic Disease Care: Mixed Methods Systematic Review and Thematic Synthesis.Journal of medical Internet research · 2026Pooled it
- Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis.International dental journal · 2026Pooled it
- The Effectiveness of Artificial Intelligence in Undergraduate Health Professions Education: Systematic Review and Meta-Analysis of Randomized Controlled Trials.JMIR medical education · 2026Pooled it
- The 5R's of large language model-assisted diagnosis: A practical framework for hospitalists.Journal of hospital medicine · 2026Article
- Performance of large language models in ambiguous treatment scenarios of early breast cancer.Breast (Edinburgh, Scotland) · 2026Article
- Generative AI and Clinicians Show Comparable Prognostic Reasoning From Clinical Narratives in Biologic-Treated CRSwNP.World journal of otorhinolaryngology - head and neck surgery · 2026Article
- Article
- Comparing the text-based diagnostic reasoning performance of emergency medicine physicians and large language models in both definitive and differential diagnoses using standardized clinical vignettes: a preliminary study.International journal of emergency medicine · 2026Article
- Predictors of Trust and Engagement in Personalized Healthcare: A Study of AI-Driven Diagnosis and Treatment in Saudi Arabia.Healthcare (Basel, Switzerland) · 2026Article
- Diagnostic accuracy and citation integrity of four large language models on otolaryngology vignettes.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Retrieval-Augmented Language Models for Clinical Decision Support in the Classification of Inborn Errors of Immunity.Journal of clinical immunology · 2026Article
- Large language models approach clinician performance in ESC cardiovascular risk stratification: a vignette-based benchmark study.European heart journal. Digital health · 2026Article
- How to meaningfully evaluate AI in clinical medicine.Nature medicine · 2026Article
- Hidden failure modes of large language models in healthcare-associated infection surveillance: a structured evaluation using NHSN definitions.Infection control and hospital epidemiology · 2026Article
- AI-Powered Clinical Decision Support in Dentistry: Comparative Evaluation of Large Language Models for Oral Medicine and Periodontal Diagnosis.Medical science monitor : international medical journal of experimental and clinical research · 2026Article
- Large language models approach specialist-level accuracy in dental pain diagnosis: a comparative evaluation of four models.Journal of dental anesthesia and pain medicine · 2026Article
- A systematic review of the limitations of large language models in generating healthcare content.PLOS digital health · 2026Article
- Large Language Models Using Clinical Text in Pediatrics: A Scoping Review.JAMA network open · 2026Article
- Performance Comparison of a Neuro-Symbolic Large Language Model System Versus Human Experts in Acute Cholecystitis Management.Journal of clinical medicine · 2026Article
- Decoding AI Competence: Benchmarking Large Language Models (LLMs) in Ovarian Cancer Diagnosis and Treatment-A Systematic Evaluation of Generative AI Accuracy and Completeness.Diagnostics (Basel, Switzerland) · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Background: With the rapid development of artificial intelligence (AI) technology, especially generative AI, large language models (LLMs) have shown great potential in the medical field. Through massive medical data training, it can understand complex medical texts and can quickly analyze medical records and provide health counseling and diagnostic advice directly, especially in rare diseases. However, no study has yet compared and extensively discussed the diagnostic performance of LLMs with that of physicians. Objective: This study systematically reviewed the accuracy of LLMs in clinical diagnosis and provided reference for further clinical application. Methods: We conducted searches in CNKI (China National Knowledge Infrastructure), VIP Database, SinoMed, PubMed, Web of Science, Embase, and CINAHL (Cumulative Index to Nursing and Allied Health Literature) from January 1, 2017, to the present. A total of 2 reviewers independently screened the literature and extracted relevant information. The risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST), which evaluates both the risk of bias and the applicability of included studies. Results: A total of 30 studies involving 19 LLMs and a total of 4762 cases were included. The quality assessment indicated a high risk of bias in the majority of studies, primary cause is known case diagnosis. For the optimal model, the accuracy of the primary diagnosis ranged from 25% to 97.8%, while the triage accuracy ranged from 66.5% to 98%. Conclusions: LLMs have demonstrated considerable diagnostic capabilities and significant potential for application across various clinical cases. Although their accuracy still falls short of that of clinical professionals, if used cautiously, they have the potential to become one of the best intelligent assistants in the field of human health care.
Indexed as
Identifiers
What Socratic holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the Socratic graph.