AI Agent Detection Requires Three-Class Framework, Not Binary Bot Classification
An arXiv preprint (2607.26935) reveals that existing bot detection systems are ineffective at recognizing AI agents that navigate through automation, categorizing all traffic simply as human or bot. The authors introduce a three-class detection model that differentiates between humans, bots, and AI agents. In a controlled benchmark, a binary MLP classifier incorrectly identified 39.1% of genuine AI agents as human, while a SAINT binary transformer misclassified 34.5%. By incorporating a distinct agent category, they achieved a perfect per-class agent F1 score of 1.000 over 30 iterations (3 model families × 10 seeds). Additionally, the research outlines a five-tier evasion ladder, which includes passive observation and GAN-generated paths (n = 2299 evasions), arguing that the confusion stems from the limitations of binary classifiers in representing AI agents.
Key facts
- arXiv preprint 2607.26935 proposes three-class detection for humans, bots, and AI agents.
- Binary MLP misclassifies 39.1% of AI agents as human.
- Binary SAINT transformer misclassifies 34.5% of AI agents as human.
- Three-class framework achieves per-class agent F1 = 1.000 in all 30 runs.
- Five-level evasion ladder includes GAN-generated trajectories and real human cursor replay.
- Binary classifiers structurally cannot represent AI agent traffic.
- Study uses controlled benchmark with 3 model families and 10 seeds.
- Evasion dataset includes 2299 real human cursor data points.
Entities
Institutions
- arXiv