Multi-Modal LLMs for Multi-Page Handwritten Document Transcription

A recent study published on arXiv (2502.20295) explores the application of multi-modal large language models (MLLMs) in transcribing handwritten documents that span multiple pages. The research tackles the difficulties associated with handwriting text recognition (HTR), highlighting that current methods often necessitate fine-tuning on labeled datasets, which can be impractical, or depend on zero-shot solutions like OCR engines and MLLMs. Although MLLMs have potential as end-to-end transcribers and OCR post-processors, there is a lack of empirical studies assessing various prompting techniques for HTR, particularly for documents with multiple pages. The paper emphasizes that most handwritten texts are multi-page and share contextual elements like semantic content and handwriting style, yet MLLMs are generally applied at the page level, overlooking this shared context. It also notes that MLLMs are typically utilized as either text-only post-processors or image-only OCR solutions, rather than integrating multiple modes. The research introduces a range of methods that combine OCR, LLM post-processing, and MLLM prompting to enhance transcription accuracy. This paper was marked as a replace-cross on arXiv, signifying a revision, and is pertinent to fields such as digital humanities, archival digitization, and AI-assisted transcription of historical documents.