Chess-trained language models rely on pattern-matching, not understanding
A recent study disputes the assertion that language models trained on chess data possess a true understanding of the game. Researchers developed KinGPT, a character-level model with 25 million parameters, which excelled against the 3 billion-parameter ChessGPT on a suite of 600 mate-in-N puzzles and the 4 billion-parameter C1-4B on a 20-theme puzzle benchmark. The study's authors contend that the impressive results stem primarily from pattern recognition rather than actual comprehension. Additionally, they present LLM-Modulo, a framework that incorporates a verifier-in-the-loop, which enhanced RedPajama 3B's accuracy for best moves from 1.2% to 21.2% and the validity of move generation from 19.3% to 95.3% on mate-in-N puzzles, achieving results comparable to ChessGPT.
Key facts
- KinGPT is a 25M-parameter character-level language model trained only on (position, best-move) pairs.
- KinGPT exceeded 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite.
- KinGPT outperformed 4B-parameter C1-4B on a 20-theme puzzle benchmark.
- The study asserts that impressive benchmark performance is largely explained by pattern-matching.
- LLM-Modulo is a verifier-in-the-loop framework.
- LLM-Modulo raised RedPajama 3B's best move accuracy from 1.2% to 21.2%.
- LLM-Modulo improved move generation validity from 19.3% to 95.3% on mate-in-N puzzles.
- The gains from LLM-Modulo are comparable to those achieved from ChessGPT.
Entities
Institutions
- arXiv