Alibaba Field Experiment Shows Generative AI Boosts Service Speed but Not Objective Quality

In a significant field experiment in partnership with Alibaba, researchers investigated how a generative AI assistant influenced the performance of e-commerce after-sales customer service. Human agents engaged in digital chat support were randomly given access to the AI, which generated diagnoses and proposed solutions at the beginning of interactions. Agents had the freedom to accept, alter, or ignore the AI's recommendations, enabling the assessment of both AI access and actual usage effects. Results indicated that generative AI enhanced service speed and subjective quality, as reflected in customer ratings, though it did not significantly impact objective quality, measured by customer retrials. The improvements stemmed from more than just automation; the AI transformed agent-customer dynamics, leading to quicker responses and a more proactive approach from agents, while customers contributed less. These behavioral shifts continued into later stages of chats. The research, highlighting that generative AI can boost efficiency and perceived quality without necessarily minimizing follow-up needs, was published on arXiv with the identifier 2603.29888.