Domain-level metacognitive monitoring varies widely across 33 frontier LLMs

A recent study published on arXiv (2605.06673) investigated metacognitive monitoring across 33 advanced LLMs from eight different model families. Researchers evaluated 1,500 MMLU items, distributing 250 items per domain across six distinct areas. The analysis calculated the Type-2 AUROC for each model-domain combination based on verbalized confidence scores (ranging from 0 to 100), resulting in 47,151 data points. Findings indicated that all models demonstrating above-chance aggregate monitoring had significant variation at the domain level. The Applied/Professional knowledge domain was the most easily monitored (mean AUROC = .742, ranking in the top-2 for 21 out of 33 models), whereas Formal Reasoning and Natural Science were the most challenging (one of the two ranked in the bottom-2 for 27 out of 33 models). The middle three domains showed no statistical distinction (Kendall's W = .164). A coherence analysis at the subject level (within-domain similarity ratio = 0.95) validated the six-domain grouping as a useful benchmark taxonomy rather than an established latent construct.