LLMs and SLMs Fail Security Compliance in Terraform Generation

A recent arXiv study (2608.02672) assesses the performance of seven AI models in generating secure Infrastructure-as-Code (IaC) for AWS Terraform. This research incorporates Checkov and Trivy security scanners within a GitLab CI/CD pipeline, analyzing three closed LLMs—Claude Opus 4, GPT-5.4, and Gemini 2.5 Pro—and four open SLMs—Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, and Magicoder-S-CL-7B—across 17 scenarios. Utilizing two prompting strategies, the study evaluates performance at three security levels using pass@5. Notably, it finds that syntactic validity and security compliance are largely independent; for example, WizardCoder-33B has a 77.8% validation rate but no Checkov compliance, while Claude Opus 4 achieves 23.1% Checkov and 92.5% Trivy pass rates with detailed prompts. The findings emphasize that prompt engineering alone cannot guarantee security, necessitating a focus on security evaluation and innovative IaC generation methods.