CoREB: A Multitask Benchmark and Model for Code Search Beyond Retrieval

A new benchmark for code retrieval and reranking, named CoREB, has been developed by researchers, along with a specialized code reranker. This benchmark overcomes the shortcomings of current code search assessments by encompassing the entire process, which includes reranking and queries styled after developer needs. CoREB is derived from counterfactually rewritten problems from LiveCodeBench across five programming languages, released in stages with graded relevance assessments. The study evaluated eleven embedding models and five rerankers over three tasks: text-to-code, code-to-text, and code-to-code. Findings indicate that embeddings tailored for code excel in code-to-code retrieval, while reranking enhances overall task performance. The initiative aims to minimize data contamination, label inaccuracies, and binary relevance issues in code search evaluations.