GROVE Visualization Tool Reveals Hidden Distributional Structure in Language Model Outputs

GROVE, a novel interactive visualization tool, tackles the challenges of assessing language models based solely on individual outputs. Created following a formative study with 13 researchers utilizing language models, GROVE illustrates multiple generations as intersecting paths within a text graph. This method uncovers distributional structures, including modes, rare edge cases, and sensitivity to minor prompt alterations that are usually obscured when users engage with single completions. The tool underwent evaluation in three crowdsourced studies with 47, 44, and 40 participants aimed at complementary distribution analysis. The research paper was published on arXiv under identifier 2604.18724v1. Current workflows often falter as users tend to over-generalize from anecdotes while refining prompts for open-ended tasks. GROVE maintains access to raw outputs while highlighting shared structures, branching points, and clusters in language model outputs.