Three-Layer Dogfooding Framework for AI Chat Agents with NPC Simulation

A novel evaluation framework for AI chat agents has been proposed in a paper on arXiv (ID 2608.09939), filling a significant void in quality assurance for teams implementing LLM-based chat systems. Named 'three-layer dogfooding,' this framework integrates three distinct testing approaches: Layer 1 utilizes a canonical question bank for fundamental response evaluations, Layer 2 incorporates random-walk multi-turn dialogues to mimic user interactions, and Layer 3 features a goal-directed NPC (Non-Player Character) simulator, which includes five goal types and a ten-category failure taxonomy. A longitudinal case study spanning approximately three months analyzed a production multi-agent system with 257 evaluation runs and a suite of 108 scenarios. Findings indicate that the three layers yield complementary regression signals, revealing weak cross-layer correlation for response quality within synchronized runs (Spearman rho ranging from -0.15 to 0.14) and negative correlation across the longitudinal series. This indicates that each layer assesses different facets of agent performance, allowing the framework to uncover failures that single-method evaluations may overlook. The research offers a practical strategy for teams to ensure that real users can successfully achieve their objectives in multi-turn conversations, a vital component of chatbot implementation.