Experimentation
Experiments and Gradient Descent
How to find the best version via iterative experiments via analogy to gradient descent
Good Metrics Maximize Surprise
“Let it surprise you.”
May 2026
Estimating Demand via Experiments (when They won’t let you randomize price...)
A more customer-friendly way to experiment on price
September 2025
“It Takes Too Long To Experiment”
But this is wrong!
July 2025
Experimentation is a reasonable tax.
Experimentation helps us reach a socially efficient equilibrium in an understudied market.
March 2025
High Velocity Metrics!
How to choose high-velocity/data-driven/other buzzwords metrics in a simple, automate-able way.
February 2025
Instead Of Testing For Sample Ratio Mismatch (SRM), Correct For It
Don’t Test, Correct: Robust Experiment Estimation
Use Decision Theory to Choose Significance Levels for Experiments
Alpha shouldn’t always be 5%.
January 2025
Does It Matter That Online Experiments Interact?
What interactions do, why they are just like any other change in the environment post-experiment, and some reassurance
Scale Experiment Decision-Making with Programmatic Decision Rules
Decide what to do with experiment results in code
In Defense of Statistical Significance
We have to draw the line somewhere
October 2024
Losing Is The Value Of Experimentation
The value of experimentation is in the losses. You ran the experiment because you thought it was a good idea.