AgentPulse: Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment

A new framework called AgentPulse has been developed by researchers to continuously evaluate 50 AI agents across ten different workload categories. This system utilizes four key factors: Benchmark Performance, Adoption Signals, Community Sentiment, and Ecosystem Health, which integrate 18 real-time indicators sourced from GitHub, package registries, IDE marketplaces, social media, and benchmark leaderboards. An analysis of these agents indicates that the four factors provide largely complementary insights, with the strongest correlation (ρ=0.61) observed between Adoption and Ecosystem factors. Furthermore, a circularity-controlled assessment of 35 agents revealed that the Benchmark+Sentiment sub-composite, excluding signals from GitHub, effectively predicts external adoption metrics like GitHub stars (ρ_s=0.52, p<0.01) and Stack Overflow question volume (ρ_s=0.49, p<0.01). This framework overcomes the limitations of static benchmarks that only assess capabilities at a single moment, failing to reflect real-world adoption or maintenance.