Reasoning Effort in API Contracts: Higher Cost, No Proven Accuracy Gain
A registered paired contrast of Sonnet 5 indicated that including a high reasoning-effort term in an API contract increases the average cost per call by $0.01031 compared to excluding it, with a confidence interval ranging from +$0.00204 to +$0.01974. The research utilized 30 AIME 2026 items and five calls for each item, revealing no significant difference in accuracy; the accuracy difference was +0.0133, with an interval of -0.0267 to +0.0467, suggesting a potential increase of up to 4.67 percentage points. API purchasers acquire a dated contract that encompasses the requested model, the reasoning-effort term or its absence, output rail, service product, prompt, and pricing schedule. The cost per correct answer under the explicit-high contract was $0.08665.
Key facts
- The study compared Sonnet 5 with explicit high reasoning effort against the same model with effort omitted.
- The experiment used 30 AIME 2026 items with five calls per item.
- Mean delivered cost was $0.01031 per call higher with the explicit-high contract (95% CI +$0.00204 to +$0.01974).
- The accuracy contrast was +0.0133 (95% CI -0.0267 to +0.0467), showing no detected accuracy difference.
- The interval allows a possible accuracy gain of up to 4.67 percentage points that the design cannot rule out.
- API contracts include model, reasoning-effort term, output rail, service product, prompt, and price schedule.
- Cost per correct answer under the high-effort contract was $0.08665.
Entities
Institutions
- arXiv
- Sonnet 5
- AIME 2026