Under the Null

Under the Null

Home
Notes
The Experimentation Posts
Archive
About

Experimentation

“It Takes Too Long To Experiment”
But this is wrong!
 • Zach Flynn
Experimentation is a reasonable tax.
Experimentation helps us reach a socially efficient equilibrium in an understudied market.
 • Zach Flynn
High Velocity Metrics!
How to choose high-velocity/data-driven/other buzzwords metrics in a simple, automate-able way.
 • Zach Flynn
Instead Of Testing For Sample Ratio Mismatch (SRM), Correct For It
Don’t Test, Correct: Robust Experiment Estimation
 • Zach Flynn
Use Decision Theory to Choose Significance Levels for Experiments
Alpha shouldn’t always be 5%.
 • Zach Flynn
Does It Matter That Online Experiments Interact?
What interactions do, why they are just like any other change in the environment post-experiment, and some reassurance
 • Zach Flynn
Scale Experiment Decision-Making with Programmatic Decision Rules
Decide what to do with experiment results in code
 • Zach Flynn
In Defense of Statistical Significance
We have to draw the line somewhere
 • Zach Flynn
Losing Is The Value Of Experimentation
The value of experimentation is in the losses. You ran the experiment because you thought it was a good idea.
 • Zach Flynn
The Best Way To Do Sample-Sizing For An Experiment
If you want to check out an easy package for the approach to sample-sizing in this blog, see https://samplesizer.com (can’t believe I snagged that…
 • Zach Flynn
Peeking Not Considered Harmful
You may have received the impression that it is bad to “peek” at experiment results before your experiment finishes.
 • Zach Flynn
Making Social Decisions from Experiments: Convincing the Skeptic
How do we decide what to do given that we ran an A-B experiment with success metric Y?
 • Zach Flynn
© 2026 Zach Flynn · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture