LLMs as Credibility Annotators in Danish Asylum Decisions
A new study investigates the use of large language models (LLMs) for automating text annotation in a specialized legal domain: identifying credibility assessments in Danish asylum decision texts. The researchers introduce RAB-Cred, a Danish text classification dataset with expert annotations, annotator confidence, and case outcomes. They benchmark 21 open-weight models and 30 prompt combinations for zero-shot and few-shot classification, analyzing error patterns, inter-class confusion, and correlation with human annotations. The work highlights challenges for underrepresented languages and domains requiring subtle expert understanding.
Key facts
- Study evaluates LLMs for credibility assessment annotation in Danish asylum decisions.
- Introduces RAB-Cred dataset with expert annotations and metadata.
- Benchmarks 21 open-weight models and 30 prompt combinations.
- Focuses on zero-shot and few-shot classification.
- Analyzes error consistency and inter-class confusion.
- Addresses underrepresented languages and specialized domains.
- Published on arXiv with ID 2605.13412.
- Task involves identifying presence and sentiment of credibility assessments.
Entities
Institutions
- arXiv
Locations
- Denmark