Workbook Time Machine: New Benchmark for LLM Spreadsheet Creation
A new tool called the workbook time machine has been developed by researchers to create benchmarks for assessing the abilities of language models in generating spreadsheet elements such as formulas and charts. When applied to public workbook datasets, it generates wtmcorpus, which consists of triples (input workbook, output workbook, query) across four types of artifacts. The team also established wtmbench, an evaluation benchmark comprising 150 tasks with queries categorized into three levels of specificity. Findings indicate that factors like query specificity, agent orchestration, and the interface API play a crucial role in the performance of LLMs on Excel-related tasks. This research is presented in the paper 'Back to the Future: A workbook time machine for spreadsheet creation benchmarks,' submitted to arXiv in the Computer Science > Artificial Intelligence category, accessible at https://arxiv.org/abs/2608.07873.
Key facts
- The workbook time machine is a pipeline that automatically creates benchmarks for spreadsheet creation tasks.
- It evaluates language models on creating formulas, charts, pivot tables, and conditional formatting.
- Applied to public workbook corpora, it produces wtmcorpus, a collection of (input workbook, output workbook, query) triples.
- wtmbench is a 150-task evaluation benchmark curated from wtmcorpus.
- Queries in wtmbench are at three levels of specificity.
- Evaluations show that query specificity, agent orchestration, and interface API significantly affect LLM performance on Excel tasks.
- The paper is titled 'Back to the Future: A workbook time machine for spread sheet creation benchmarks'.
- The paper is available on arXiv under Computer Science > Artificial Intelligence.
Entities
Institutions
- arXiv