← Research / Experiment Design

Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

Read full paper →
AuthorsNicholas Larsen, Alex Deng, Jiheng Zhang, Ron Kohavi, Nathaniel Stevens
JournalThe American Statistician
Year2024
DOI10.1080/00031305.2023.2257237
Citations70

What Problem It Solves

Provides a canonical overview or reference point for the relevant DoOperator research area.

What problem it solves

Provides a canonical overview or reference point for the relevant DoOperator research area.

How it works

A modern review of statistical methodology for online controlled experiments, including sensitivity, heterogeneity, long-term effects, optional stopping, and interference.

When to use it

Use when orienting a new paper, blog post, benchmark, or research plan in this area.

Limitations and failure modes

Do not cite the overview as evidence that a specific method works in a specific deployment setting without checking the underlying primary paper.

Read full paper →More Experiment Design

Related papers

Paper

The Design of Experiments

Ronald A. Fisher · 1935

Paper

Controlled experiments on the web: survey and practical guide

Ron Kohavi, Roger Longbotham, Dan Sommerfield +1 more · 2009

Paper

On Causal Inference in the Presence of Interference

Eric J. Tchetgen Tchetgen, Tyler J. VanderWeele · 2012

Paper

Always Valid Inference: Bringing Sequential Analysis to A/B Testing

Ramesh Johari, Leo Pekelis, David J. Walsh · 2015