Write a PREreview

Life as a Function: Why Transformer Architectures Struggle to Gain Genome-Level Foundational Capabilities

by Hassan Hassan, Kyle Puhger, Ali Saadat, and Maximilian Sprang

Posted: January 14, 2025
Server: bioRxiv
DOI: 10.1101/2025.01.13.632745

Recent years have seen a flurry of generative nucleotide models, mostly of limited utility. In this paper, we use the functional representation of DNA as a complex, composite function on the plane of evolution to extend the theoretical unification of ecological and evolutionary change to the problem of synthetic DNA models. Through experiments on synthetic and real DNA sequences we show that next-token prediction decoder transformer architectures are limited in their capacity to learn such functions.

You can write a PREreview of Life as a Function: Why Transformer Architectures Struggle to Gain Genome-Level Foundational Capabilities. A PREreview is a review of a preprint and can vary from a few sentences to a lengthy report, similar to a journal-organized peer-review report.

Before you start

We will ask you to log in with your ORCID iD. If you don’t have an iD, you can create one.