Random Survey-Country Labels Still Bias LLM Social Inference

A recent study published on arXiv (2608.06085) examines whether revealing that a survey-country label is assigned at random diminishes its impact on social inference by large language models (LLMs). The researchers performed an audit within records across five fixed API models, six countries, and seven targets selected for development. They analyzed results when the source of the random label was either hidden or presented as uniform and independent of the record. In their main post-review analysis of 72 records, both types of labels resulted in country-direction shifts of 0.214, with a paired attenuation of 0.0003 (95% CI [-0.0157, 0.0166]), indicating no notable reduction. A confirmed survey country decreased Brier loss by 0.040 (95% CI [0.024, 0.056]), while the regret from random labels included zero. A separate mixed-coverage consistency panel showed continued positive movement for disclosed-random labels and verified utility. These results imply that merely informing users about the randomness of a label does not lessen its bias on LLM predictions, presenting a significant challenge for AI transparency.