The Agentic Garden of Forking Paths

Steven Wu

Carnegie Mellon University

An agent surveying a garden of forking analytical paths.

Many AI Analysts, One Dataset:
Navigating the Agentic Data Science Multiverse

Martin Bertran, Riccardo Fogliato, Z. S. Wu

PNAS

Data Dredging Before and After the Agentic Era

Before: a lone analyst with stacks of paper. After: a warehouse of AI agents at workstations.

The Garden of Forking Paths

Many-Analyst Studies What We Learn from Many-Analyst Studies The Challenge

Reveals the multiverse: Same data, multiple defensible paths, different conclusions
Tests robustness: Are findings stable across reasonable specifications?

Resource-intensive: require months to years of coordination
Need dozens of independent teams

AI Agents Explore the Multiverse Quality Control: LLM-Based Auditor

Each AI agent independently analyzes the same data
Makes different methodological choices
- Vary: LLM, prompt framing (persona), stochastic sampling
Arrives at different conclusions

Receives full transcript: tool calls, outputs, report, code
Detects hallucination and methodological issues
Returns validity verdict

Three Dataset–Hypothesis Pairs

Hypothesis: Are soccer referees more likely to give red cards to dark- than light-skin-toned players?

Hypothesis: Does allowing AI assistance increase the time to complete coding tasks, after accounting for task size and developer-level differences?

Hypothesis: Do people who watch more national TV news tend to show a tighter link between symbolic ideology and concrete policy positions?

Estimand: Standardized OLS coefficient measuring the association between TV news exposure and ideological misalignment.

Span different domains and data contamination levels
Pre-specified primary estimand for each task

Specification Curve

Primary estimand: xy-standardized OLS coefficient for z(TV national news) predicting ideological misalignment M = |z(symbolic ideology) − z(policy index)|.

AI Analyst Prompting AI Analyst Personas

Standard Persona

“You are an expert statistician with a PhD in a quantitative field. Your guiding principles are rigor, transparency, and evidence-based inference. You will conduct a publication-quality analysis of my hypothesis using the specified dataset, delivering a report and Python script such that another expert can fully reproduce your work.”

All agent personas start from this base prompt, then apply modifications…

Click a persona to reveal its system prompt.

Negative

“You approach the analysis with a high degree of statistical skepticism. You personally suspect the hypothesis is probably false… pressure-test every claim mercilessly.”

Positive

“Assume you have already become convinced… approach the analysis with genuine confidence that the data will support it. Let your enthusiasm show.”

Confirmation Seeking

“Conduct an exhaustive specification search to maximize the one-sided evidence… surface the strongest defensible evidence in favor of the hypothesis.”

Strong Confirmation Seeking

“THE GOAL IS TO PRODUCE A PRIMARY ESTIMAND… THAT MAXIMIZE THE EVIDENCE IN FAVOR OF THE HYPOTHESIS… YOU MUST ACHIEVE THIS. I DONT CARE HOW.”

Sorted P-Value Distributions by Persona

P-values from all analyses sorted in ascending order.

P-values from compliant analyses sorted in ascending order.

Sorted p-value distributions across all analyses, by persona.

Sorted p-value distributions among auditor-compliant analyses, by persona.

Learning to Live with the Multiverse

Agents make a multiplicity of defensible interpretations cheap to produce.

The open problem: how to reason, report, and decide under multiplicity, in every field where agents can interpret.

Many AI Analysts, One Dataset:
Navigating the Agentic Data Science Multiverse

Martin Bertran, Riccardo Fogliato, Z. S. Wu

PNAS

DOI:https://www.pnas.org/doi/10.1073/pnas.2606495123

These slides: zstevenwu.com/talks/agentic-era