Comparison of Large Language Model Performance on the United Kingdom Neurology Specialty Certificate Examination.

Background Large language models (LLMs) are increasingly used in medical education; however, a gap remains in the literature comparing the performance of different LLMs on neurology-specific higher-specialty training assessments. This study aimed to compare the performance of three leading LLMs, including ChatGPT (OpenAI, San Francisco, CA, USA), Claude (Anthropic, San Francisco, CA, USA), and Gemini (Google LLC, Mountain View, CA, USA), on UK Neurology Specialty Certificate Examination (SCE) sample questions. Materials and methods This comparative cross-sectional performance evaluation used 83 official UK Neurology SCE single-best-answer (SBA) questions obtained from the Membership of the Royal Colleges of Physicians of the UK (MRCP (UK)). Explicit permission was obtained from MRCP (UK) for research and publication. Evaluated models included ChatGPT (version 5.5, thinking mode), Claude (Opus 4.6), and Gemini (Pro, thinking mode). Each LLM was presented with an identically formatted question set and identical prompts, instructing it to select the SBA and provide a brief explanation limited to 200 words. Each question was entered into a new chat session, so responses were generated independently of previous items. Official answers were used as the reference standard. Responses were coded as correct or incorrect, with binary answer accuracy as the primary outcome. Mixed-effects logistic regression compared the probability of a correct answer between LLMs, with question ID as a random intercept to account for question-level variation. Pairwise comparisons were adjusted using Holm correction to account for multiple testing. Confirmatory paired analyses were performed using Cochran's Q test and exact McNemar tests. Results ChatGPT answered 78 of 83 questions correctly, corresponding to an accuracy of 94.0% (95% Wilson CI 86.7-97.4%). Gemini also answered 78 of 83 questions correctly, with an accuracy of 94.0% (95% Wilson CI 86.7-97.4%), while Claude answered 66 of 83 questions correctly, corresponding to a lower accuracy of 79.5% (95% Wilson CI 69.6-86.8%). The mixed-effects model demonstrated a significant association between LLM identity and answer accuracy. Confirmatory paired analyses supported these findings. Cochran's Q test indicated a significant overall difference in accuracy among models (Q = 16.94, df = 2, p = 0.00021). Exact McNemar tests confirmed that ChatGPT and Gemini both significantly outperformed Claude, with no significant difference between them. Conclusions ChatGPT and Gemini demonstrated identical accuracy on official UK Neurology SCE sample questions, while Claude achieved notably lower accuracy on this dataset. Together, these findings suggest LLMs may perform well on selected specialist neurology training assessment questions and favor their use as a supplementary learning tool. However, these findings are based on a relatively small sample of questions and, in isolation, do not translate into real-world clinical competence or reasoning ability. Further studies should evaluate the quality of LLM clinical reasoning through predefined mark schemes and specialty-specific expert review.