ArticleHealthcare (Basel, Switzerland)2026
Evaluating the Quality of Artificial Intelligence-Generated Information on Cleft Lip and Palate: A Comparative Cross-Sectional Study.
Article in Healthcare (Basel, Switzerland), 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
BACKGROUND/
objectivesLarge language models (LLMs) are increasingly consulted for information about cleft lip and palate (CLP), yet the reliability of their outputs across clinical domains has not been evaluated. This study aimed to compare the quality of CLP-related information generated by GPT-4o and Gemini 2.5 Pro across multiple thematic domains using a validated quality instrument and a reliability-first analytic framework.
methodsFifty-four standardized CLP questions across six domains were submitted to GPT-4o (OpenAI) and Gemini 2.5 Pro (Google DeepMind) on 25 September 2024 via their public interfaces, using new, history-free sessions and default settings, yielding 108 responses. Three independent, CLP-experienced raters scored each response using the Global Quality Score (GQS; 1-5 scale assessing accuracy, completeness, and clinical usefulness). Before comparing models, we applied a reliability-first filter: only domains where all three raters showed substantial agreement (Fleiss' kappa [κ] ≥ 0.60) were included in statistical comparisons. Domains that failed this threshold were analyzed qualitatively to identify the source of disagreement. A descriptive taxonomy of errors was developed for low-scoring responses.
resultsThree domains met the reliability threshold (General Care Information, General Cleft Information, and Pre-Treatment Information; 30 paired questions). Both models performed at a high and practically equivalent level: GPT-4o median GQS 4.33 (IQR 4.00-5.00) versus Gemini 2.5 Pro 5.00 (IQR 4.00-5.00); the difference was not statistically significant (Wilcoxon V = 139.00,
conclusionsBoth models provided high-quality CLP information in domains supported by clinical consensus, indicating they may serve as useful adjuncts for general patient and family counseling. Clinicians should, however, verify any treatment-specific content against current institutional protocols before relaying it to patients. Future research should assess readability, alignment with health literacy, and patient comprehension of AI-generated CLP information.
Indexed as
Identifiers
What Socratic holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the Socratic graph.