A transformer-based language model reveals developmental constraint and network complexity during zebrafish embryogenesis.

Understanding how regulatory complexity and constraint shape organismal development remains a central challenge in biology. The developmental hourglass framework posits that mid-embryogenesis-the phylotypic stage-is a period of heightened conservation and coordinated regulatory organization. We test this hypothesis using Zebraformer, a transformer-based language model trained on single-cell transcriptomic data from zebrafish embryos. Zebraformer learns context-sensitive representations that capture temporal progression, anatomical identity, and regulatory relationships, yielding gene and cell embeddings that recapitulate the developmental axis and increasing transcriptional divergence over time. In contrast, attention-derived gene networks reveal a transient reorganization of regulatory architecture during the phylotypic stage, marked by tightly coordinated gene modules, reduced cross-module connectivity, and diminished local redundancy. Sensitivity to perturbation emerges specifically when regulatory interaction structure is taken into account, rather than from perturbation magnitude alone, highlighting that constraint during this stage is embedded in network topology rather than representational fragility. These findings are supported by graph-theoretic metrics and gene ontology enrichment analyses. Together, our results refine the hourglass framework by localizing developmental constraint to the architecture of gene regulatory networks and demonstrate that language models can extract interpretable biological structure from high-dimensional single-cell data.