Comparative evaluation of seven large language models in providing home phototherapy care guidance for neonatal hyperbilirubinemia.

Background: Neonatal hyperbilirubinemia affects 60-80% of term newborns. The American Academy of Pediatrics (AAP) recommends that infants with elevated bilirubin after discharge who meet specific criteria can receive home phototherapy to preserve family bonding. Despite caregivers increasingly seeking online health information, no research has evaluated large language models (LLMs) as auxiliary guidance tools for home phototherapy. This study aimed to comparatively evaluate seven mainstream LLMs as auxiliary educational tools for guiding caregivers in home phototherapy management, identifying which models provide reliable operational guidance versus those presenting safety risks. Methods: Seven high-ranking LLMs were selected: ChatGPT-4o, Copilot, Gemini 1.5 Flash, Claude 3.5 Sonnet, DeepSeek-R1, GLM-4, and ERNIE 4.0 Turbo. We developed a 15-item questionnaire covering seven domains based primarily on AAP neonatal hyperbilirubinemia guidelines. Each LLM received three priming prompts followed by 15 caregiver-perspective questions. Two investigators independently queried each model (November 2024-February 2025). Three board-certified neonatologists blindly evaluated 210 responses using modified Likert scales across five dimensions: accuracy, completeness, reproducibility, empathy, and readability. Expert clinician responses served as the "gold standard" and benchmarks. Results: DeepSeek-R1 demonstrated superior performance across all dimensions (total score: 19.31/20), with 87.8% of responses rated as "completely correct" for accuracy (4.89±0.12) and completeness (4.89±0.11). ChatGPT-4o ranked second (16.73/20), with accuracy (4.22±0.72) and completeness (4.18±0.74) scores significantly higher than those of other models. Claude 3.5 Sonnet and GLM-4 achieved moderate performance (>4.0 for clinical dimensions), whereas Copilot, Gemini 1.5 Flash, and ERNIE 4.0 Turbo showed suboptimal accuracy. Conclusions: DeepSeek-R1 and ChatGPT-4o achieved near-expert accuracy, potentially reliable for basic queries under oversight. Claude 3.5 Sonnet and GLM-4 showed moderate performance with notable gaps. Copilot, Gemini, and ERNIE demonstrated critical errors. However, all models exhibited hallucinated references and a lack of patient-specific reasoning. LLMs should not replace professional medical judgment in treatment decisions. Safe implementation requires restricting high-performing LLMs to narrowly defined procedural questions only, with mandatory physician verification.