CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
Samenvatting
CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation. arXiv:2602.20571v1 Announce Type: new Abstract: Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification-formulating a valid research design under stated assumptions-and estimation-implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 138 real-world datasets, curated from 85 peer-reviewed research papers and four widely-used causal-inference textbooks. For each query a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 84 % of cases, full identification-specification correctness drops to only 30 %, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.
Waarom dit ertoe doet
Deze technologische ontwikkeling kan de manier waarop AI in de zorg wordt ingezet fundamenteel veranderen.
Context (AI-duiding)
Klik op “Toon context” om AI-duiding op te halen.
Nieuwsbrief
Wekelijks dit soort signalen in je inbox
De nieuwsbrief bundelt nieuwe signalen, relevante verschuivingen en korte duiding zodat je minder afhankelijk bent van incidentele sitebezoeken.
Scores
De mate waarin dit signaal de Nederlandse gezondheidszorg kan beïnvloeden (1 = minimaal, 5 = transformatief).
Hoe snel actie of aandacht nodig is (1 = kan wachten, 5 = onmiddellijke aandacht vereist).
De mate van onzekerheid over de uitkomst of timing (1 = zeer voorspelbaar, 5 = zeer onzeker).
Tags
Bronnen
- CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and EstimationarXiv - Artificial Intelligence (cs.AI) —
Pipeline versie: 0.2.0 | Gegenereerd door: pipeline