Color Bias in Vision Language Models: Stealth Visual Prompts Shift Sentiment

A recent study published on arXiv (2608.14286) explores the impact of visual styling biases, especially color, on vision language models (VLMs) when they analyze text depicted in images. The authors present 'Stealth Visual Prompts,' which discreetly modify visual features such as color and contrast while maintaining the original meaning. Results reveal that highlighting positive words in green consistently skews sentiment predictions positively, suggesting that VLMs often overlook visual styling, resulting in biased conclusions. This finding poses risks for industrial applications that utilize VLMs, including recruitment tools and recommendation systems. The research meticulously examines how visual styling influences VLM performance and investigates the effects of these changes on latent representations in the vision encoder. The paper, titled 'Seeing Red, Thinking Bad: Color Bias in Vision Language Models,' is authored by researchers and available on arXiv.