Under the Null

Under the Null

Home
Notes
The Experimentation Posts
Weekly Round-Ups
I Want Zach's Help
Archive
About

Experimentation

Experiments and Gradient Descent
How to find the best version via iterative experiments via analogy to gradient descent
 • Zach Flynn
Good Metrics Maximize Surprise
"Let it surprise you."
 • Zach Flynn
Estimating Demand via Experiments (when They won't let you randomize price...)
A more customer-friendly way to experiment on price
 • Zach Flynn
“It Takes Too Long To Experiment”
But this is wrong!
 • Zach Flynn
Experimentation is a reasonable tax.
Experimentation helps us reach a socially efficient equilibrium in an understudied market.
 • Zach Flynn
High Velocity Metrics!
How to choose high-velocity/data-driven/other buzzwords metrics in a simple, automate-able way.
 • Zach Flynn
Instead Of Testing For Sample Ratio Mismatch (SRM), Correct For It
Don’t Test, Correct: Robust Experiment Estimation
 • Zach Flynn
Use Decision Theory to Choose Significance Levels for Experiments
Alpha shouldn’t always be 5%.
 • Zach Flynn
Does It Matter That Online Experiments Interact?
What interactions do, why they are just like any other change in the environment post-experiment, and some reassurance
 • Zach Flynn
Scale Experiment Decision-Making with Programmatic Decision Rules
Decide what to do with experiment results in code
 • Zach Flynn
In Defense of Statistical Significance
We have to draw the line somewhere
 • Zach Flynn
Losing Is The Value Of Experimentation
The value of experimentation is in the losses. You ran the experiment because you thought it was a good idea.
 • Zach Flynn
© 2026 Zach Flynn · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture