JMed48k: Japanese Medical Licensing Benchmark for VLMs

A new benchmark called JMed48k has been developed by researchers to assess vision-language models in the context of Japanese healthcare licensing. This dataset, derived from official PDF documents from Japan's Ministry of Health, Labour and Welfare, comprises 48,862 exam questions and 20,142 images sourced from 11 national licensing exams conducted between 2005 and 2025, with visual elements categorized using an 8-type taxonomy. Additionally, a subset named JMed48k-Eval features 12,484 scored questions from the past five years, including 9,905 text-only questions and 2,579 with images. The research team analyzed 21 models—both proprietary and open-source—reporting their performance on text-only and image-inclusive queries. A paired image-removal audit assessed four states of answer transition by comparing questions with and without images.