Startups Rethink Transformer Architecture to Build Faster, Cheaper LLMs

A fresh wave of startups is taking on the supremacy of transformers, the neural network design that powers leading large language models (LLMs). Since their introduction by Google in 2017, transformers have been hindered by significant computational demands and challenges with lengthy contexts. Companies such as Subquadratic, Manifest AI, Liquid AI, Inception, and Pathway are creating alternatives, including sparse attention mechanisms, power retention strategies, and diffusion-based text generation. Subquadratic’s SubQ competes with popular LLMs, while Manifest AI offers rolling summaries to condense context. Liquid AI merges transformers with liquid neural networks for better energy efficiency. Inception's Mercury 2 outpaces GPT-4 by tenfold, and Pathway's Dragon Hatchling has successfully tackled over 97% of difficult sudoku puzzles. OpenAI is expected to allocate $50 billion for computing in 2023.