ArticleNature communications2025
Benchmarking large language models for biomedical natural language processing applications and recommendations.
Article in Nature communications, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 82 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
82 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Precision phage therapy in the AI/ML era: a systematic review of discovery-to-clinical translation evidence.Frontiers in microbiology · 2026Pooled it
- Benchmarking commercial large language models for gene-disease-phenotype extraction from full-text human genetics literature.Quantitative biology (Beijing, China) · 2026Article
- Augmenting structured diagnoses through effective use of pre-trained large language models on clinical notes.JAMIA open · 2026Article
- Systematic assessment of text summarization methods for biomedical literature from frequency methods to language models.iScience · 2026Article
- A multi-site benchmarking framework for scalable extraction of geriatric care constructs from electronic health records.npj health systems · 2026Article
- Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease-Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study.JMIR formative research · 2026Article
- Patient-Centered Summarization Framework for AI Clinical Summarization: Mixed Methods Study.Journal of medical Internet research · 2026Article
- Article
- SynBioGPT2: A dynamic reasoning framework enables high-fidelity design of microbial cell factories.Biodesign research · 2026Article
- Article
- Searching the druggable genome using large language models.Bioinformatics (Oxford, England) · 2026Article
- Coala: a standard-based framework for converting CWL-described command-line tools into agentic toolsets.Bioinformatics (Oxford, England) · 2026Article
- A machine-assisted framework for systematic error analysis in clinical concept extraction.Nature communications · 2026Article
- An automatic consult reply system for therapeutic plasma exchange using retrieval-augmented generation.Vox sanguinis · 2026Article
- Causal graph neural networks for healthcare.Nature biomedical engineering · 2026Review
- Tutorial: guidance on the use of large language models for medical research.Nature protocols · 2026Review
- A systematic evaluation and benchmarking of text summarization methods for biomedical literature: From word-frequency methods to language models.bioRxiv : the preprint server for biology · 2026Article
- Review
- MedADL: high-throughput information extraction of functional status from electronic health records to advance frailty assessment in older adults.The journals of gerontology. Series A, Biological sciences and medical sciences · 2026Article
- CellExLink: End-to-end cell-type recognition and normalization in biomedical text.PLoS computational biology · 2026Article
22 more citing papers are in PubMed but not listed here.
Corrections and comments
- Update of
Authors and funding
21 authors.
Funding
Abstract
The rapid growth of biomedical literature poses challenges for manual knowledge curation and synthesis. Biomedical Natural Language Processing (BioNLP) automates the process. While Large Language Models (LLMs) have shown promise in general domains, their effectiveness in BioNLP tasks remains unclear due to limited benchmarks and practical guidelines. We perform a systematic evaluation of four LLMs-GPT and LLaMA representatives-on 12 BioNLP benchmarks across six applications. We compare their zero-shot, few-shot, and fine-tuning performance with the traditional fine-tuning of BERT or BART models. We examine inconsistencies, missing information, hallucinations, and perform cost analysis. Here, we show that traditional fine-tuning outperforms zero- or few-shot LLMs in most tasks. However, closed-source LLMs like GPT-4 excel in reasoning-related tasks such as medical question answering. Open-source LLMs still require fine-tuning to close performance gaps. We find issues like missing information and hallucinations in LLM outputs. These results offer practical insights for applying LLMs in BioNLP.
Indexed as
Identifiers
What Socratic holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the Socratic graph.