Transformers Trained as Universal Computers via MicroPy Programs

A study reveals that a compact transformer can be developed into a universal computer by learning to run programs in MicroPy, a streamlined yet computationally complete programming language. Utilizing PENCIL scaffolding, the transformer predicts execution in small steps, allowing for efficient processing within a limited context window. Following training on randomly generated, nonsensical MicroPy scripts, the model successfully adapts to human-created programs, including bit manipulation, binary arithmetic, and SAT problem-solving. The trained model exhibits out-of-distribution generalization, assessing new programs from previously unseen distributions. Given that MicroPy can represent any computation, these findings offer empirical support that a standard transformer can be trained to serve as a universal computer. This paper is available on arXiv in the computer science and artificial intelligence sections.