ArticleJMIR medical informatics2024
Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study.
Article in JMIR medical informatics, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 67 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
67 citing papers in PubMed.
- Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus-based bone stress injury.Journal of experimental orthopaedics · 2026Article
- Ambient Artificial Intelligence Scribes: A Scoping Review With Implications for Otolaryngology.The Laryngoscope · 2026Article
- Article
- Article
- Turing problems in otolaryngology: a scoping review of the principal challenges of artificial ıntelligence and large language models.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Review
- Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care.Critical care explorations · 2026Article
- A Self-Controlled Benchmark of Retrieval-Augmented Generation for Large Language Models on Clinical Guideline Questions.Diagnostics (Basel, Switzerland) · 2026Article
- AI hallucinations in academic writing: implications for research integrity.Naunyn-Schmiedeberg's archives of pharmacology · 2026Review
- Saudi Medical Students' Perceptions and Attitudes of Integrating Generative Artificial Intelligence Integration in Medical Education: A Cross-Sectional Study.Health science reports · 2026Article
- Investigating the Potential Effects of Medical AI Systems on Physician Autonomy: Pretest of a Semistructured Qualitative Interview Guide.JMIR formative research · 2026Article
- Microlearning in medical histology: a longitudinal evaluation of faculty-created five-minute videos using five theoretical frameworks and Kirkpatrick assessment.BMC medical education · 2026Article
- Artificial stupidity or logimorphism? How misuse of language warps our thinking about 'artificial intelligence'.European heart journal. Digital health · 2026Article
- A reproducible statistical evaluation framework for large-sample assessment of AI-generated medical references: cross-platform application of the reference hallucination score.BMC medical research methodology · 2026Article
- Enhancing the quality and trustworthiness of large language model-generated summaries of clinical oncology literature.JAMIA open · 2026Article
- Child and adolescent psychiatry: challenges, solutions, opportunities, and future directions.World psychiatry : official journal of the World Psychiatric Association (WPA) · 2026Article
- Where Evidence-Based Medicine Meets AI: Promise, Pitfalls, and Practice.AEM education and training · 2026Article
- Article
- Methodological Considerations for AI-Powered Chatbots in Oral Cancer Awareness: Stakeholder Diversity, Instrument Transparency, and Safety Auditing.Clinical and translational science · 2026Article
- Eliminating Nonvalue-Added Tasks in Health Care: Artificial Intelligence in the Orthopedic Office.Journal of hand surgery global online · 2026Article
- Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability.PLOS digital health · 2026Article
7 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundArtificial intelligence (AI) chatbots have recently gained use in medical practice by health care practitioners. Interestingly, the output of these AI chatbots was found to have varying degrees of hallucination in content and references. Such hallucinations generate doubts about their output and their implementation.
objectiveThe aim of our study was to propose a reference hallucination score (RHS) to evaluate the authenticity of AI chatbots' citations.
methodsSix AI chatbots were challenged with the same 10 medical prompts, requesting 10 references per prompt. The RHS is composed of 6 bibliographic items and the reference's relevance to prompts' keywords. RHS was calculated for each reference, prompt, and type of prompt (basic vs complex). The average RHS was calculated for each AI chatbot and compared across the different types of prompts and AI chatbots.
resultsBard failed to generate any references. ChatGPT 3.5 and Bing generated the highest RHS (score=11), while Elicit and SciSpace generated the lowest RHS (score=1), and Perplexity generated a middle RHS (score=7). The highest degree of hallucination was observed for reference relevancy to the prompt keywords (308/500, 61.6%), while the lowest was for reference titles (169/500, 33.8%). ChatGPT and Bing had comparable RHS (β coefficient=-0.069; P=.32), while Perplexity had significantly lower RHS than ChatGPT (β coefficient=-0.345; P<.001). AI chatbots generally had significantly higher RHS when prompted with scenarios or complex format prompts (β coefficient=0.486; P<.001).
conclusionsThe variation in RHS underscores the necessity for a robust reference evaluation tool to improve the authenticity of AI chatbots. Further, the variations highlight the importance of verifying their output and citations. Elicit and SciSpace had negligible hallucination, while ChatGPT and Bing had critical hallucination levels. The proposed AI chatbots' RHS could contribute to ongoing efforts to enhance AI's general reliability in medical research.
Indexed as
Identifiers
What Socratic holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the Socratic graph.