AssertLLM2: New Benchmark for LLM-Based Hardware Assertion Generation
AssertLLM2 has been launched by researchers as an open-source benchmark aimed at assessing large language models (LLMs) in their ability to create SystemVerilog Assertions (SVAs) from design specifications. This benchmark overcomes the shortcomings of current benchmarks by offering realistic task definitions, organized design specifications, and thorough evaluations. It encompasses 83 actual hardware designs categorized into 13 functional groups, each featuring a verified golden RTL alongside systematically altered buggy versions. AssertLLM2 facilitates two practical scenarios: bug-prevention, where assertions are crafted from specifications to avert errors, and bug-hunting, where assertions assist in detecting bugs within RTL code. This initiative seeks to streamline assertion-based verification, a vital yet labor-intensive aspect of hardware design.
Key facts
- AssertLLM2 is an open-source benchmark for LLM-based assertion generation.
- It contains 83 real-world designs across 13 functional categories.
- Each design includes a structured specification, golden RTL, and buggy RTL variants.
- The benchmark supports bug-prevention and bug-hunting settings.
- It addresses limitations of existing benchmarks in task formulation and evaluation.
- The work aims to automate SystemVerilog Assertion generation.
- Assertion-based verification is a cornerstone of modern hardware design.
- The benchmark is described in arXiv:2605.27472.
Entities
Institutions
- arXiv