LEX-EC: A New Framework for Auditing LLM Personality Classification
A novel black-box audit framework named LEX-EC has been developed by researchers to assess zero-shot LLM personality classification. This framework integrates prevalence and agreement diagnostics alongside controlled lexical ablation, allowing for the differentiation of marginal-distribution effects from trait-related signals that can be retrieved with limited evidence. Through LEX-EC, the researchers demonstrated that various text genres present distinctly different profiles: free-form essays show the widest yet weak signal; in graduate student introductions, the Extraversion link diminished post-masking; and single Facebook statuses provide minimal stable evidence, suggesting a potential content or length threshold. Masking demographic and topical elements reduced some associations while maintaining others via function words and emotional terms. This framework aims to bridge the interpretability gap in LLM personality classification, providing a means to audit models in black-box environments. The paper can be found on arXiv under ID 2607.24435.
Key facts
- LEX-EC is a reusable black-box audit framework for LLM personality classification.
- It combines prevalence and agreement diagnostics with controlled lexical ablation.
- The framework distinguishes marginal-distribution effects from trait-associated signal.
- Free-form essay text contains the broadest but still weak signal.
- In graduate student introductions, an observable Extraversion association weakened after masking.
- Single Facebook statuses yield little stable evidence even in a trait-balanced sample.
- Masking topical and demographic content weakened some associations while leaving others detectable from function words and affective terms.
- The paper is available on arXiv with ID 2607.24435.
Entities
Institutions
- arXiv