PatRe Benchmark Models Full Patent Examination Lifecycle

Researchers have unveiled PatRe, the inaugural benchmark designed to simulate the complete patent examination process, which encompasses the generation of Office Actions and the responses from applicants. This examination process is intricate and involves multiple stages that necessitate both technical knowledge and legal reasoning, facing increasing pressure from the growing number of applications. Previous benchmarks primarily treated patent examination as either discriminative classification or static extraction, overlooking its interactive and iterative aspects, akin to the peer review and rebuttal processes in academic publishing. PatRe consists of 480 authentic cases and accommodates both oracle and retrieval-simulated evaluation environments. It reconceptualizes patent examination as a dynamic, multi-turn dialogue of justification and counterargument. Comprehensive experiments with various large language models (LLMs) provide valuable insights into their performance, highlighting distinctions between proprietary and open-source models.