Scaling Autoregressive Transformer for Single-Cell Generation
A study published on arXiv (2608.02961) presents a self-supervised task aimed at generating single-cell gene expression vectors. Utilizing a causal transformer alongside a learned quantized VAE tokenizer, the model is trained using cross-entropy loss. The research explores the biological accuracy of the generated vectors and the scaling behavior of the pretraining loss. By adjusting various parameters and datasets, the authors discover the initial jointly-fit two-exponent scaling law and the compute-optimal frontier. The full paper can be accessed at https://arxiv.org/abs/2608.02961.
Key facts
- The paper is on arXiv with ID 2608.02961.
- The task is self-supervised generation of single-cell gene expression vectors.
- The model is a causal transformer with a quantized VAE tokenizer.
- Training uses cross-entropy loss.
- Evaluation compares generated distribution to ground truth.
- Scaling properties are studied by varying parameters and data.
- The paper reports the first jointly-fit two-exponent scaling law.
- A compute-optimal frontier is identified.
Entities
Institutions
- arXiv