TSQueryBench: New Benchmark Reveals Asymmetry in LLM Generation and Evaluation
TSQueryBench serves as a synthetic benchmark featuring 500 instances of time series data across 10 distinct query types, each accompanied by correct, partially correct, and incorrect explanations. The evaluation involved six large language models focusing on explanation generation, relative ranking, independent scoring, and multi-anomaly detection. A significant discovery indicates that these models can effectively assess even when they do not produce accurate explanations. To evaluate explanations, it is essential to verify numerical assertions against structured data. TSQueryBench offers ground-truth labels for a thorough assessment of both generation and evaluation. This research emphasizes that a model's capabilities in judging and explaining may not be in sync, which is crucial for implementing LLM evaluators in sectors such as finance, healthcare, and climate science. The paper can be found on arXiv under the identifier 2604.02118.
Key facts
- TSQueryBench is a controlled synthetic benchmark with 500 time series instances.
- The benchmark covers 10 query types.
- Each instance is paired with correct, partially correct, and incorrect explanations.
- Six large language models were evaluated.
- Four tasks: explanation generation, relative ranking, independent scoring, multi anomaly detection.
- Central finding: consistent generation evaluation asymmetry.
- Evaluation of time series explanations requires verifying numerical claims against structured data.
- The paper is available on arXiv with identifier 2604.02118.
Entities
—