CangjieBench: New Benchmark for LLMs on Low-Resource Programming Language

A new benchmark called CangjieBench has been developed by researchers to assess large language models (LLMs) using Cangjie, a general-purpose programming language with limited resources. This benchmark features 248 carefully translated samples from HumanEval and ClassEval, encompassing both text-to-code and code-to-code tasks. The investigation evaluates various LLMs across four scenarios: direct generation, syntax-constrained generation, retrieval-augmented generation (RAG), and agent. Findings reveal that direct generation yields unsatisfactory results, while syntax-constrained generation strikes an optimal balance between accuracy and computational efficiency. This research fills a void in previous studies, which largely concentrated on domain-specific languages (DSLs), neglecting the challenges faced by general-purpose languages. The results underscore the difficulties LLMs encounter with low-resource languages and propose syntax-constrained generation as a viable enhancement strategy.