Fisher-R1: LLM Agent for Reliable Hypothesis Testing

The recently developed Fisher-R1 is an open-weight LLM agent aimed at tackling the challenge of unreliable hypothesis testing in scientific research. This agent automates hypothesis testing by analyzing datasets, generating code, and conducting comprehensive analyses. However, researchers discovered that LLM agents often commit subtle inferential mistakes, resulting in erroneous conclusions even when analyses are executed correctly. To bridge this gap, they created P-Bench, a benchmark featuring 425 realistic, open-ended hypothesis-testing tasks across fields like economics, biology, and medicine. Each task requires the agent to choose a statistical method, calculate a p-value, and reach a conclusion based solely on a scientific hypothesis and dataset. Fisher-R1 is designed for rigorous hypothesis testing, with its model weights publicly accessible. This research is documented in a paper on arXiv (2608.07437) under the 'new' announcement type, emphasizing the necessity for benchmarks that evaluate the statistical validity of reported p-values, a shortcoming not addressed by current benchmarks.