Evaluating LLM Trade-offs for Enterprise Automation: Workflow Generation Lessons

A recent study published on arXiv (2608.03311) explores insights gained from assessing six large language models aimed at generating AI-driven workflows within a production enterprise platform. This research evaluates these models across 29 practical IT automation scenarios, utilizing two distinct generation pipeline architectures and conducting eight independent runs for each setup, culminating in a total of 2,784 runs. The original monolithic pipeline recorded structural success rates ranging from 31.5% to 82.8%, with many models facing challenges in generating complex JSON. A newly designed piecewise pipeline breaks down workflow construction into smaller tasks, enhancing performance. The study emphasizes the need for enterprise compliance management to swiftly adapt to changing regulatory standards like DORA, AI RMF, and FedRAMP, as well as stringent remediation SLAs. Traditional static orchestrators frequently struggle in hybrid cloud settings, where event-driven evaluations necessitate automation code that can adjust to runtime contexts in mere seconds. The paper sheds light on the trade-offs associated with employing LLMs for automation, underscoring the significance of pipeline architecture and model choice.