QuantumKatas Adapted from Q# to Qiskit for LLM Benchmarking

Researchers have transformed Microsoft's QuantumKatas by transitioning it from Q# programming to Qiskit. They developed a comprehensive benchmark comprising 350 tasks across 26 distinct categories aimed at assessing large language models (LLMs). The tasks range from basic quantum gates to complex algorithms like Grover's, Simon's, and Deutsch-Jozsa, also addressing error correction, key distribution, and quantum games. Each task is designed with a natural language prompt, a verified answer, and involves deterministic testing through classical simulation. The evaluation encompassed 16 LLMs using 7 different prompting strategies, resulting in a total of 39,200 model assessments.