LLMs Show Positional Bias in Semantic Sensitivity Testing Framework

An experimental framework capable of scaling has been developed to methodically assess the sensitivity of LLMs to minor semantic variations in document comparisons, likened to finding a needle in a haystack. Researchers incorporated single sentences with semantic alterations within a broader context across tens of thousands of document pairs. They evaluated five LLMs while experimenting with different types of perturbations, such as negations, conjunction swaps, and changes to named entities. The context types included original content compared to topically irrelevant material, varying the position of the needle and the length of documents. The findings indicated that LLMs demonstrate a positional bias within documents, differing from previously observed candidate-order effects, with most models imposing stricter penalties on earlier semantic differences. When altered sentences were placed within unrelated contexts, similarity scores consistently dropped. This framework is outlined in arXiv preprint 2604.18835v1, recognized as a cross-type publication, offering a multifaceted perspective on how LLMs handle subtle semantic changes in text comparison tasks.