A Guideline-Concordant Chatbot Framework for Structured Colorectal Cancer Screening: Multistage Feasibility Study.

backgroundColorectal cancer (CRC) screening relies on structured risk assessment and guideline-concordant communication, which remain challenging to implement consistently in real-world practice. Digital tools based on large language models (LLMs) may support such workflows, but their feasibility and safety in structured screening contexts have not been well evaluated.

objectiveThis study aimed to develop and evaluate a guideline-concordant chatbot framework for structured CRC screening.

methodsA multistage feasibility study was conducted. In phase 1, baseline performance of contemporary LLMs was assessed using 14 standardized CRC screening questions and validated expert-rated instruments. In phase 2, structured prompt versions were iteratively optimized based on screening guidelines and expert feedback and tested using simulated user scenarios. In phase 3, the optimized chatbot was evaluated in 50 screening-eligible adults to assess feasibility, safety, and guideline concordance.

resultsIn phase 1, both LLMs demonstrated satisfactory performance in responding to standardized CRC screening questions, with DISCERN instrument for AI-generated content scores of 12.02 (SD 0.30) for GPT-4o and 13.36 (SD 0.26) for DeepSeek-V3, Global Quality Score scores of 3.96 (SD 0.10) for GPT-4o and 4.39 (SD 0.08) for DeepSeek-V3, Natural Language Assessment Tool for AI scores of 21.41 (SD 0.34) for GPT-4o and 22.73 (SD 0.27) for DeepSeek-V3, and Patient Education Materials Assessment Tool adapted for AI outputs scores of 0.900 (SD 0.016) for GPT-4o and 0.906 (SD 0.015) for DeepSeek-V3. In phase 2, iterative prompt optimization significantly improved all 6 expert-rated dialogue evaluation dimensions (all P values <.001). In phase 3, the optimized chatbot successfully collected complete CRC screening risk information and generated guideline-concordant screening recommendations for all participants (N=50). No unsafe or inappropriate outputs were identified.

conclusionsThis study demonstrated the feasibility, preliminary safety, and guideline-concordant performance of a structured chatbot framework for CRC screening communication under the study conditions. Beyond patient education, the proposed framework may support key components of the CRC screening workflow, including risk information collection, risk stratification, and generation of guideline-concordant screening recommendations. Further prospective studies are needed to evaluate the framework's impact on patient-centered outcomes, screening uptake, and clinical effectiveness.