Evidence map›Paper›PMID 40121370›Full record

ArticleNPJ digital medicine2025

A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians.

Hirotaka Takita, Daijiro Kabata, Shannon L Walston, Hiroyuki Tatekawa, Kenichi Saito, Yasushi Tsujimoto, Yukio Miki, Daiju Ueda

Registry-linked trialAbstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in NPJ digital medicine, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT07632859 (Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes), which is not on this map. Cited by 106 papers, 6 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
106citing papers in PubMed, 6 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

NCT07632859 completednot on this mapstarted 2026, after this paper: background citation

Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes: A Retrospective Study of 600 Cases

TypeobservationalSponsorMarmara University Pendik Training and Research HospitalRan2026 to 2026Enrolled600ConditionsEmergency Medicine, Diagnostic Errors, Artificial Intelligence (AI) in Diagnosis
3 · Its place in the literature

Who cites it

106 citing papers in PubMed, 6 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Trial
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Review
  16. Article
  17. Review
  18. Article
  19. Article
  20. Review

46 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

8 authors.

Hirotaka TakitaDepartment of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan.
Daijiro KabataCenter for Mathematical and Data Science, Kobe University, Kobe, Japan.
Shannon L WalstonDepartment of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan.ORCID http://orcid.org/0000-0002-7268-8313
Hiroyuki TatekawaDepartment of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan.
Kenichi SaitoCenter for Digital Transformation of Health Care, Graduate School of Medicine, Kyoto University, Kyoto, Japan.
Yasushi TsujimotoOku Medical Clinic, Osaka, Japan.
Yukio MikiDepartment of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan.
Daiju UedaDepartment of Diagnostic and Interventional Radiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, Japan. ai.labo.ocu@gmail.com.ORCID http://orcid.org/0000-0002-3878-3616

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

While generative artificial intelligence (AI) has shown potential in medical diagnostics, comprehensive evaluation of its diagnostic performance and comparison with physicians has not been extensively explored. We conducted a systematic review and meta-analysis of studies validating generative AI models for diagnostic tasks published between June 2018 and June 2024. Analysis of 83 studies revealed an overall diagnostic accuracy of 52.1%. No significant performance difference was found between AI models and physicians overall (p = 0.10) or non-expert physicians (p = 0.93). However, AI models performed significantly worse than expert physicians (p = 0.007). Several models demonstrated slightly higher performance compared to non-experts, although the differences were not significant. Generative AI demonstrates promising diagnostic capabilities with accuracy varying by model. Although it has not yet achieved expert-level reliability, these findings suggest potential for enhancing healthcare delivery and medical education when implemented with appropriate understanding of its limitations.

Identifiers

PMID40121370
PMCPMC11929846

What Socratic holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the Socratic graph.