Evaluating large language models for diabetic retinopathy multiple-choice question generation in clinical ophthalmic education.

Background: Large language models (LLMs) are increasingly used in medical education, but their ability to generate ophthalmic multiple-choice questions (MCQs) remains unclear. Diabetic retinopathy (DR), a core ophthalmic training topic, provides a framework for evaluating LLM-based item generation. Methods: Five publicly accessible LLMs completed 60 predefined DR MCQ tasks under a standardized Chinese single-turn prompt and blueprint, yielding 300 items. Evaluation included structural completeness, format compliance, keyed-answer accuracy, textual features, response time, and blinded expert ratings across six educational domains. Because the same 60 tasks were completed by all five models, between-model comparisons were performed using paired task-level analyses. Continuous and ordinal outcomes were compared using Friedman tests, followed by Bonferroni-corrected paired Wilcoxon signed-rank tests when appropriate. Inter-rater reliability was assessed using intraclass correlation coefficients, and Spearman analyses examined associations between output features and expert-rated quality. Results: Between-model differences were observed in all textual variables and response time (all Friedman test Conclusion: All five LLMs generated structurally complete and format-compliant DR MCQ drafts, but differences remained in factual accuracy, expert-rated item quality, output style, and usability. Gemini 3 and ChatGPT-5.4 showed the most favorable balance between correctness and expert-rated usability, supporting LLMs as assisted item-generation tools rather than replacements for expert review.