PTXBench Benchmark Evaluates LLMs for GPU Kernel Optimization with Architecture-Specific PTX
PTXBench, introduced in an arXiv preprint, benchmarks large language models for GPU kernel optimization using architecture-specific PTX. It measures functional correctness, runtime instruction execution, and speedup against frontier libraries on GEMM and attention workloads using H100 and B200 GPUs. Results show uneven capability across models, with notably poor performance on complex attention backward tasks; executing target instructions does not guarantee competitive speed. No model consistently matches frontier libraries. The authors adapted Qwen3.6-27B via supervised fine-tuning, finding repair-conditioned training improves some tasks, but generalization remains uneven, influenced by data coverage, balance, and reasoning-teacher quality. The benchmark offers an audit for future work.
Key facts
- PTXBench is a new benchmark for evaluating and adapting LLMs for GPU kernel optimization.
- It focuses on architecture-specific PTX instructions.
- Evaluation covers functional correctness, runtime execution of target instructions, and speedup.
- Workloads include GEMM and attention on H100 and B200 GPUs.
- Success rates drop significantly on complex attention backward workloads.
- Executing target instructions does not automatically lead to competitive performance.
- No evaluated model consistently outperforms or matches frontier libraries across the suite.
- Adaptation used supervised fine-tuning on Qwen3.6-27B, with repair-conditioned training improving some tasks but generalization uneven.
Entities
—