TRIAGE Framework Tests LLMs' Metacognitive Control Under Token Budgets

A novel assessment framework named TRIAGE examines the ability of large language models to manage a queue of tasks within a limited token budget, a type of metacognitive control previously analyzed in human cognition. This framework necessitates that models adhere to a single, organized strategy that determines the order of problems to tackle, the sequence of actions, and the token distribution for each, all without receiving execution feedback. Plans are evaluated against an oracle that possesses complete knowledge of each problem's solvability and associated costs, resulting in a triage efficiency ratio. The research assesses both frontier and open-source models, with and without reasoning capabilities, on competitive math problems. Findings reveal that existing LLMs struggle with effective prospective metacognitive control, frequently misprioritizing solvable tasks and inefficiently allocating tokens. This research underscores a significant deficiency in the capabilities of autonomous agents and implies that future enhancements may necessitate architectural modifications or training focused on metacognitive techniques.