Test Syntax Structure Affects AI Code Generation Quality

An extensive empirical investigation utilizing the SEGA framework analyzed over 830 generated files, 12 models, and 3 providers. The findings indicate that inline test syntax, specifically Python doctests, achieves nearly flawless preservation at 100% and a correctness rate ranging from 92% to 100% for AI code generation. In contrast, separated test syntax, such as Rust #[test] blocks, reveals significant discrepancies in model performance, with correctness varying from 0% to 100%, highlighting the lack of correlation between preservation and correctness. This research, available on arXiv (2604.19826), evaluates inline against separated test formats on a d-ary heap implementation, noting that model performance changes over generations, with one model significantly failing to suppress tests.