ExtractBench: New Benchmark for Schema-Guided Document Extraction
A new evaluation framework called ExtractBench has been launched to improve the assessment of schema-guided extraction processes in business documents. Featured in a recent arXiv paper, it measures crucial aspects like accuracy and completeness while considering different costs. ExtractBench includes 4,869 pages from 370 documents, representing 67 categories and eight distinct business sectors. The evaluation process combines consensus from automated systems for real documents, validated values for synthetic samples, and human checks for forms. It provides metrics for order-insensitive value accuracy and grounding, alongside preliminary performance insights for commercial vision-language models, although details remain limited.
Key facts
- ExtractBench is a benchmark for schema-guided extraction.
- It is the first to score value accuracy, record completeness, grounding, and cost together.
- The dataset includes 4,869 pages from 370 enterprise documents.
- It covers 8 business domains and 67 document types.
- The curation pipeline uses independent-system agreement, known values, and human verification.
- Metrics include order-insensitive value F1 and grounding F1 at word and page levels.
- Commercial VLMs are evaluated on this benchmark.
- The paper is available on arXiv with ID 2607.29677.
Entities
Institutions
- arXiv