Today, we’re breaking from this blog’s usual topic (statistics, economics, etc) to indulge in a far more important topic: a hobby project I built seven years ago. Aside from being a trip down memory lane, it’s possible this blog has a point. Two of them, in fact:

1. Doing things just because is a perfectly good reason to do things, and

2. Extensible double-entry accounting apps are a valid form of artistic expression.

Subscribe now

It was the Fall of 2018, and I was to be wed in the Spring. I did not have much in the way of savings in those ancient days, and weddings cost money. I needed to budget for it.

Naturally, I needed to write an extensible double-entry accounting system to do this. It had to be programmable in Scheme, of course. I cannot overemphasize how much all of these requirements were unavoidable design constraints. It is impossible to budget for a wedding without Lisp.

And so, leas was born. Leas is an extensible accounting system that you can use interactively in the terminal. It’s extensible with Scheme, a full programming language, embedded in the app.

[I wrote it without thinking about LLMs—it was 2018—but now that we are in the new world, these tools do make it even more useful. “Codex, write me a script in Guile Scheme to update my home value based on whatever Zillow says it is and record the incremental value as home equity gains” just works like magic. Try getting a web app to do that.]

The initial idea was to make a tool that fit in between two kinds of accounting tools.

1. ledger. A tool that produces reports from specially-formatted text files. It’s very UNIX-y. I like ledger. It is highly extensible because you just need a script that modifies a text file. Easy. I actually first toyed with the idea of making leas an interactive tool to modify ledger files, but I didn’t like the result, and I didn’t find ledger very convenient to use in its natural state. I wanted something more interactive that was still extensible.

2. GUI-tools like GNUCash. I used GNUCash in grad school, but it was a separate GUI app that I’d always forget to open. I wanted something quick and easy from the terminal. GNUCash had some extensibility (in Scheme, too) but not super easy to use.

I also just thought the idea of an Emacs-like personal finance tool was neat-o.

So, I came up with this little structure:

• Interactive C-based command line app.

• The prompt was just a Scheme interpreter that called various Scheme/C functions exposed to Scheme.

My first critical design decision: I realized right away that I did not want to type parentheses all the time:

(command arg1 arg2)

So, I made the interpreter insert the outer parenthesis automatically so that this Just Works (tm):

command arg1 arg2

But, I wanted the program to be more… interactive. I didn’t want to have to type full commands. So, I settled on a UX like this:

> spend
5: Groceries
6: Rent
...
To Account: 5
1: Checking
...
From Account: 1
Amount: 109.23
Description: Walgreens
Year: 2026
Month: 6
Day: 22
>

Of course, I could just code this interaction in C, but I wanted to make the interactive system sort of “universal” and something you could use in Scheme extensions to add new interactive commands. So I came up with this system for making any Scheme function interactive, and also added some special data types like this:

(define add-record
  (lambda ()
    (leas/call "leas/at"
               (list
                (cons "Account" "current_account")
                (cons "Amount" "real")
                (cons "Description" "string")
                (cons "Day" "day")))))

So you can use leas/call to provide arguments to any non-interactive function using data types like “real” and “string” but also with some leas specific datatypes like “day” (for a date), “current_account” for the currently selected account, “account” for an account, “expense_account” (for an Expense account), “liability_account” (for a Liability account), and so on. So now we have some actual accounting data types.

This little structure let me write a small set of core routines and data types in C and then most of the interactive structure in Scheme (the UI, more or less) which was nice.

The basic structure was:

• Book

  • Account

    • Transaction

Each Account has a Type. One of:

1. Asset

2. Liability

3. Income

4. Expense

Buying a McDouble, taking out loans, paying them back, and earning money are all just transfers between accounts. You transfer funds from the Income account to the Asset account on payday. You transfer from a Liability to an Expense account when you pay with a credit card or from a Liability to an Asset account when you take out a loan.

There is this concept of the “current” account, a pointer that attaches to a certain account and marks it “active”. That makes it possible to use commands which just take the current account as their argument which makes it quicker to navigate around. For example, lt lists the last few transactions in the current account.

I use leas every day, so the thing has ended up having lots of niceties. There are commands for doing things like paying off a loan (two transfers: one from an asset account to an expense account to pay interest and one to the liability account to pay off the principal).

I’ve written Scheme scripts to automatically update the values of any funds/stocks/etc. I hold by scraping the internet.

I should point out that this whole “budgeting” thing actually worked. Go figure. Keeping track of spending did let me kind of thread the needle on the wedding. It reduced stress, increased confidence, and gave my skin a healthy glow.

The whole thing is only about 2000 lines of code. Simple, but I’ve used it daily for a long time now. I don’t commit much to the repo anymore. The software is finished, more or less. I fixed an edge case in it a couple of weeks ago. I’m averaging a commit every two years since 2020. That’s the nice part about non-SaaS software. You can finish it. Nothing changes in the environment, and it’s not like any bug is a big deal. It’s running on your machine. You can’t steal someone else’s info if I make an error in the controller.

Of course, there was really no point to this project. The number of budgeting and personal finance tools out there is endless. They are better documented, tested, and so on. I could’ve found something that worked well enough to not be worth the hours I spent on it.

But I had oodles of fun, and there’s the pride of making something useful.

I don’t know shit about fixing cars, so I have to find something else to do.

There’s this terrible piece of advice folks get. If someone’s already solved the problem, don’t re-solve it. Think about another problem worth solving. It’s bad advice. You can’t solve any problem until you learn enough about it.

The best way to learn how the wheel works is to reinvent it.

Oh, and here’s the Github for this project: leas.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

This post will be about experimentation, but there’s a wind-up.

Today, we’re going to look at a class of experimentation problems that we might call “tuning.” A problem where there is a wide variety of plausible choices for a parameter, and we’re trying to find the best one among nearly infinite plausible choices.

Common parameters to optimize include:

• Price

• Promotional Discounts

• Hyperparameters of ML models (search, recommendations, etc.)

Subscribe now

Suppose you want to maximize a function f(u,t). At any point in time, you know f(u_t, t), but you can’t evaluate f at any other points but whatever u happens to be “active”. For example, you know your current prices (u_t) and you know revenue today, but you don’t know what revenue would be at other prices.

How do we learn the best u if we can only evaluate one u at a time?

Let’s think about it as an optimization problem:

Suppose f is strictly concave in u and its gradient satisfies a Lipschitz condition (we’ve decided to be formal-ish today—suit and tie). Then, we could solve this problem via gradient descent, i.e.:

For some suitable sequence of (a).

The problem is: we don’t know the derivative of f, and we don’t know how to evaluate f at other points.

The trick is that we can use experiments to iterate on the gradient descent problem. An experiment is just like computing the objective function and its gradient. It takes a few weeks to compute, though, and our data limits how many points we can compute at once.

The starting point is the relationship between treatment effects and derivatives in this context. Suppose u is a number (not a vector), and you expose half the population to u and the other half to u+h. Well, viola, you have an estimate of the derivative:

[You can expand this to multi-dimensional u by adding more arms to the experiment or by adding structure to f that lets you learn other derivatives in a neighborhood of the current values of u you’re testing.]

First, suppose f doesn’t depend on time, and also, for the sake of simplicity, u is not a vector. Then, the optimal u doesn’t change. It’s static. A stable target you’re trying to hit. Using the gradient descent analogy, we can try to find the best u like so:

When the treatment effect from increasing u is positive, increase u. When it is negative, decrease u, and so on.

There is a vast literature on step sizes that work for gradient descent with a strictly concave f. The key property we need is a method that doesn’t require us to evaluate the function at many points because experimentation limits the number of points we can evaluate. A straightforward approach to choose a, assuming f is strictly concave, using a modern “two-point” method from the literature that works for general concave functions, c/o Malitzky and Mischenko:

Where Df_t is the current derivative estimate, and Df_t-1 is the prior derivative estimate. I’ll call this the MM-rule.

[The MM-rule formula treats dividing by 0 as infinity, so you’ll always choose the other branch of the min{*,*}.]

The main advantage of making this choice instead of the classical gradient descent step size is that you don’t need to run experiments with a ton of arms to approximate the maximization problem:

You’d operationalize this like so:

1. Initially: Run an experiment with the status quo “champion” u vs a “challenger” u + h. Choose h large enough that you’ll see some real action statistically, but small enough that you’re reasonably approximating a derivative. Choose an arbitrary a to get a new challenger. The right choice here is a bit of an art, but it’s fine to just choose it so the treatment looks “reasonable.” This is just an initial condition.

2. Do this again to have enough initial conditions to start using the MM-rule.

3. Now, run the best arm of the previous experiment versus the new challenger generated by the MM-rule.

4. Keep iterating, using the MM-rule above. Repeat step 3 forever.

The nice part about this method of experimentation is that it allows you to make careful, small but continuous iterations toward the truth instead of making large, costly attempts to jump to the optimal value (usually based on strong functional-form assumptions). You’re always testing your latest proposal against the data, so you can’t drift too far.

Practical issues you’ll run into: the formula for a might produce too small of a step size for you to see any real action. The treatments are too close together. It’s fine to put a lower bound on step size, even if it violates some guarantee of convergence. The precision the optimizers look for isn’t attainable if we’re estimating functions from noisy data.

Now, what if the function’s value changes over time, i.e., we have f(u,t)? That’s why we’d run experiments in the first place, instead of choosing different u’s and trying them out to trace out the function. We need to run experiments because otherwise we’ll confuse variation in different u’s for variation in t.

When we experiment, we learn:

So, we can trace out the derivative with respect to u from the data, as in the static function case, but the optimal u we’re chasing is changing as well.

I’m proposing using the same algorithm above to solve the problem when t changes the function (and so, the optimal u)—with a small twist.

Of course, if we’re completely general about how t changes f, this approach won’t work, because the optimal u could jump around wildly, but that’s not really what we think will happen.

Essentially, we’re going to assume that the optimal u varies “slow enough” with t that we can track it by accounting for changes in the gradient with respect to u.

The main practical change is that the MM-rule now depends on function values that we can’t just keep in memory. We need to evaluate the gradients of f(*,t), not the gradients we previously computed of f(*,t-1). So, here’s my simple, practical iteration strategy:

1. Run a four-arm experiment. The experiment has u_t (the current iteration), u_t-1 (the previous iteration), u_t + h, and u_t-1 + h (the h’s can be different if you’d like).

2. Use the results of this experiment to construct L_t.

3. Now, if you look at the formula, you’ll see that technically it depends on a_t all the way back and prior L_t’s that are function evaluations will have been evaluated on f(*,t-2), for example, instead of f(*,t). My sort of pragmatic approach here is to ignore that and use the old a_t’s where they appear as they were when you computed them.

I think this is a reasonable, practical tradeoff to avoid exploding the number of arms the experiment needs while still accounting for the fact that the function has changed, moving from t-1 to t.

Experimentation can test more than button colors. We can use it to understand the models that drive the modern biz. Models that target certain customers with promotional offers or specifically tailored marketing strategies. The number of reasonable models is vast, and finding the right one requires forcing proposed models to confront the truth about their performance. Experiments provide the ground truth and structure the search. It’s a nice pairing.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

All analyses rely on assumptions to map datasets to conclusions. The data doesn’t speak for itself. You have to shape it. Ideally, the mold you fit the data into is based on a reasonable theory of how the world works.

Subscribe now

Analysis is a trick to, with enough data, reduce the world to two potential outcomes:

1. The analysis’s conclusion is true.

2. OR: the assumptions of the analysis are wrong.

This is very useful. It moves the conversation forward. Folks can’t just disagree with your conclusions. They have to disagree with your assumptions, too.

Therefore, an analysis is persuasive only if your assumptions are more plausible than your conclusions.

If your assumptions are more plausible than your conclusions, then disagreeing with the conclusion becomes more difficult after the analysis. Before I could argue against the conclusion directly. Now, I have to argue against the assumptions, which are harder to dispute.

But if your assumptions are less plausible than your conclusions, then you haven’t convinced me. After the analysis, I know either the conclusion is true, or your assumptions are wrong. If I disagreed with the conclusion before, and it is even easier for me to disagree with your assumptions, it is just as easy to disagree with you as it was before your analysis*.

There’s a famous paper in empirical labor economics that illustrates this.

Card and Krueger (1994) claim to show no effect of a minimum wage increase on unemployment.

They do so using a difference-in-differences analysis of bordering counties in Pennsylvania and New Jersey after New Jersey raised its minimum wage. The idea is that these counties are similar to each other and face similar shocks, except that the New Jersey counties received a higher minimum wage. Card and Krueger look at changes in employment and find a small, positive, but insignificant effect. The conclusion is that the minimum wage increase did not cause unemployment. Politicians frequently cite the paper to draw the much broader conclusion that minimum wage increases don’t, more generally, cause unemployment.

It isn’t a persuasive analysis. It’s further confirmation to the believer, but it won’t convert anyone.

Suppose that, prior to the analysis, I disagree with the idea that labor demand is flat or upward-sloping. Now, after the analysis, you tell me: either you have to believe that demand is flat or upward-sloping, or you can believe that these New Jersey and Pennsylvania counties are not really comparable.

Of course, I’m going to conclude that these counties aren’t really comparable. If I have to choose between “People don’t care how much they pay for things” and “These counties aren’t comparable in some subtle way. Maybe another policy or shock that happened at around the same time wasn’t distributed evenly across New Jersey and Pennsylvania counties. Maybe these counties are just wholly incomparable because of a long history of differences in policy and polity between the two states that change how they respond to various shocks. Maybe a silly shock that we’ll never understand happened. Maybe the unemployment effect is delayed because of adjustment frictions.” All of those explanations are more credible and convincing to me than “people don’t care about how much they pay for things.”

So, the study fails to convince anyone. Proponents of a higher minimum wage like the study, but they don’t need to be convinced by it. Sometimes academic economists who find themselves in policy roles where they need to pretend to like minimum wages might find it useful as air cover to explain away past writing that says minimum wages cause unemployment. “Oh, I used to think minimum wages were bad, but then I read Card and Kreuger.” Hot take: No true believer in downward-sloping demand curves has ever been convinced by this. It’s so much easier to conclude that parallel trends don’t really hold or the post period isn’t long enough to see the effect than to invalidate a simple, powerful economic theory on the basis of a diff-in-diff.

This isn’t to argue about an economics paper from more than thirty years ago. It’s to demonstrate that:

Shocking results need mundane assumptions to convert new folks to your way of thinking.

We should be credibility-maxing?

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

*This point isn’t about Bayes’ rule. I understand how my joint prior over the assumptions and conclusions might cause my posterior beliefs about the conclusion to change, even if I view the assumption as less credible than the conclusion in terms of my prior’s marginal distributions. It’s an argument about how people are actually convinced, i.e., I don’t have a tightly constructed joint prior distribution over the likelihood of New Jersey and Pennsylvania counties being comparable and the slope of the labor demand curve. I weigh how plausible the assumptions are and how plausible I think the conclusion is, and that determines whether I end up convinced by the analysis to change my mind.

Ever since I decided to blog well, I’ve been trying to figure out the right way to write a good “links” article. All the cool kids on Substack do it, and I read the Internet. I told myself I ought to be able to do it too. Two statistics blogs and one links blog a week sounded plausible.

Like most things, though, links blogs looked so easy to get right until I tried to do it. So, we’re experimenting with formats. I think I like this one. Two Good Things. Two stories or clips that I think our worth consuming this week. We’ll get there eventually.

This one has bathhouses and AI skeptics.

Subscribe now

Bathhouses

Joe Hovde has a great blog on this company called Bathhouse. It’s a public sauna trying to become a third place—presumably, for the kinds of people who feel comfortable less-than-clothed around strangers: hot people.

A little trend. I’ve noticed that proposed Third Places end up getting accused of being cults—which is natural. They’re trying to replace the Church or the Bar. A Third Place has to have aspects of religion or addiction to satisfy the human need for connection via transcendence or dependency.

I don’t see it crossing generations, but fingers-crossed for them. That’s the key to a successful Third Place. There’s always the old guy at the Bar and the kids with fake ID’s. Church has Sunday school and funerals. Even coffee shops—when those were a bigger deal—cross generations. At least the marketing makes it look like a young single thing?

The website has an “AI Search” feature.

In any case, the article’s a great dive into the business and a category I didn’t know existed before. Check it out.

[I dig this blog in general, by the way. It’s always got some interesting dive into an industry I either don’t know much about or don’t know enough about. Good place to learn stuff.]

“[LLM Companies] Shouldn’t Be Allowed to IPO”

I thought this segment on Bloomberg with Ed Zitron where he hates on AI and AI companies was entertaining, if not persuasive.

Mr. Zitron doesn’t like AI or AI companies, but he has much stronger arguments for the Company’s IPO prospects than the Product and he kind of mixes the two willy-nilly.

The part he’s probably right about is that no one has a great business model yet for how to make money by supplying LLMs (or maybe the market structure just isn’t right yet to do it: you can’t get enough of a margin), and the set of tasks LLMs are good at is much smaller than the AI-positive discourse admits.

Folks are allowed/encouraged/threatened to use AI, and, for many tasks, the price of the tokens is still way more than the value delivered by the LLM. That’s how you get places that had never used AI until a couple years ago, burning millions of dollars to let folks spin up questionable (but expensive) dashboards without knowing SQL or save some Engineer a few (very expensive) hours.

The part he’s (super?) wrong about is this whole idea he often puts out there that the LLM product itself is bad-to-useless. There are nuanced takes on how LLMs can reduce productivity (by yours truly) because workers will substitute towards the LLMs and away from effort even for tasks where effort is more productive than LLMs, but there are other tasks where LLMs absolutely deliver strong productivity gains.

I do not want to go back to debugging without LLMs.

I spent the last weekend debugging this side project (a Rails app), and it would have taken much longer and been so much worse if it had just been me searching for bugs, writing tests, etc, instead of using Codex. I think I spent around 20 hours debugging the app with Codex (i.e. not just prompting “fix all bugs, make no mistakes” like you hear from some portions of the internet…). It easily would have taken me a few weeks without it, and it probably wouldn’t have been done to the same quality because I’m human, and I miss stuff (edge-cases, various long strings of scenarios that cause odd state X, etc).

The LLMs are a little like chess computers when it comes to debugging. Humans miss possible moves in chess all the time—even grandmasters miss mates-in-one in short time control games. Finding all possible one-shot moves on an 8x8 grid is a trivial problem for a computer, though. Similarly, LLMs can come up with ways in which someone could get something funky by hitting a certain string of endpoints much better than my feeble, meatsack mind.

That being said, it is definitely true, and this is Ed’s main point: the value of debugging code, formatting docs, etcetera does not justify the stratospheric level of capital and debt being loaded into these AI Labs. They kind of have to create God or at least replace Raytheon.

I might like (very much) being able to type “replace all my poorly-formatted math in this doc with well-formatted math,” but I’m not willing to pay much for it. I doubt it drives significant company revenue. Or, for example, with the above experience, debugging a web app:

I did it all in Codex (Free).

Yeah, so it was free for me. It wasn’t close to free for OpenAI, I’m sure. How much would I have paid? Not much. It’s just a little side project. If it was a major potential revenue stream? I guess I’d pay a thou or so for access to debugging for a month or something? But I doubt I’d have to. Competition is still too intense. I’d just substitute to Gemini or Claude. So prices stay low…

Until the shakeout stage of the industry (start here for industry lifecycle literature).

I think two things are true:

1. LLMs are super valuable.

2. LLMs are over-valued.

But they won’t both be true when the market for Anthropic’s future dividends becomes liquid and well-informed…

I hope the S-1 becomes public in time for beach reading.

Yes, we do have beaches. Lake Michigan counts.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

A/B tests don’t have to be Experiments. You can use them for other purposes. Corporate political propaganda, for example. Like when politicians cite random studies they haven’t read on a “debate” stage, the point isn’t the study itself. It’s the conclusion.

No judgment. Politics has its place.

The difference between an Experiment and Propaganda is whether you want to learn something or convince someone.

Subscribe now

If you want to convince someone, you will choose metrics that are all but guaranteed to go in a certain direction.

For example, you might remove a screen from a checkout flow and make your metric “time spent in checkout.” Of course, the time spent will go down. You already knew that before running the experiment. You’re building up a case for your preferred product direction, the one that hits your OKR. Not running an Experiment.

If you were running an Experiment, you’d want to know: what do we lose by dropping that screen? What do we gain by dropping it? What metric can I develop that lets both the positives and negatives of dropping the screen shine through, their relative light depending only on how customers behave? Maybe the extra information from the extra screen is worth nothing. Maybe it is everything. The choice of metric should not assume either. It should allow for both cases and let customer behavior decide.

In other words: Experiment metrics maximize surprise. Propaganda metrics minimize it.

Suppose you want to learn something. How do you go about finding a metric?

In the screen-drop example… Suppose the extra screen asks the customer for a bit more information about why they purchased the product. Maybe that information feeds later personalization or targeted messages to get the customer to buy again or subscribe or otherwise take the next step in their journey. It’ll always be something like this. It’s not like the extra-screen was just thrown in there willy-nilly. The folks who added it did so for a reason. We should know why before we knock it down.

We don’t want to use the probability of making it to the end of checkout because, of course, taking away extra screens helps with that.

We also don’t want the metric to be a conversion rate of users to taking the next step. Of course that’s going to fall. The extra screen makes the ad-targeting better.

What we want is a metric that could go either way.

For example, the joint probability of both completing the checkout and taking the next step—not the conditional probability.

If the conditional probability of taking the next step falls, but the number of customers who take the next step increases, then dropping the screen makes sense. You’re improving the probability of checking out without dropping the probability that a top-of-funnel visitor gets to the final stage of the journeys, even though targeting gets worse.

If these two steps have different economic models associated with them (maybe the checkout is the Basic subscription and the extra-screen provides information used to market Premium to the user), then make the metric take that into account. Give each one a dollar value and try to maximize expected dollars.

The point is to get the tradeoff in there. We’re making a move that increases the pr(checkout) and decreases the pr(premium | checkout). So we want to know whether pr(checkout, premium) = pr(checkout) x pr(premium|checkout) increases. We’re surprised either way because, while we know pr(checkout) will increase and pr(premium | checkout) will decrease, we don’t know the net effect.

So, we’ve set up our experiment metrics to learn something, and, along the way, we’ve figured out exactly what we’re uncertain about on the product-side. That’s the right way to select metrics. Like most problems, it can be solved by the algorithm:

1. Write down we know.

2. Write down what we don’t know.

3. Experiment to bridge the gap.

When I was but a wee lad learning to shoot, my dad told me the trick was to “Let it surprise you.” If we anticipate what’ll happen when we pull the trigger, we won’t hit the target.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Production functions matter. They describe how firms or people turn raw inputs into outputs, i.e., technology. They constrain what’s possible, like demand determines what’s preferable.

Subscribe now

Most empirical work on production functions in economics looks at manufacturing, agriculture, or resource production because their inputs and outputs are easier to measure. But knowledge work is the first job class to be altered by the LLMs. Codex isn’t building bridges anytime soon. So, modeling how knowledge workers produce output and how LLMs affect production matters, but I don’t have any data to work with...

We can learn a lot from theory, though…

LLMs take context (human knowledge) and use it to generate output effortlessly. Without LLMs, humans would still provide their knowledge input, but they also need to supply elbow grease: effort. This blog will show that uncertainty about the best way to do things—and the option to apply effort to counteract that uncertainty—drives how LLMs influence technology choice, production, and the returns to more accurate beliefs.

The plan is for this to be a multi-part blog series with different models of how LLMs transform human context into output.

This post treats LLMs as pure automation machines. They take instructions from a human and act just like a human would, but they save the human significant effort.

The key part of the theory is the role of “uncertainty” in how best to do tasks. It turns out to be critical for understanding the role of LLMs in knowledge production work. We’ll show the following about how uncertainty changes the role of LLMs:

• For low-uncertainty tasks, workers will use LLMs, which will reduce the time it takes to complete a task or improve the quality of their output.

• For moderate-uncertainty tasks, workers will use LLMs, which will increase the time it takes to complete a task or reduce the quality of their output. The worker’s preferred technology will not produce the best output.

• For high uncertainty tasks, workers will NOT use LLMs. Instead, they will use more effort to counteract their uncertainty.

Production without LLMs

Suppose a task can be completed in a certain amount of time, determined (without LLMs) by how smartly people decide to do the task, and the amount of effort they put in to get it done.

[You can think of “time” more abstractly as the inverse of “quality”, where lower time is equivalent to higher quality output, etc. But I’m going to stick to the strict time metaphor for ease of following the model.]

s is the best way to do a task, but it is unknown. Knowledge workers have beliefs (knowledge) about how best to do things, given by:

Without LLMs, the time it takes a knowledge worker to do a given task is given by:

Where z is the way the knowledge worker decides to do things, and e is the amount of effort they put in.

Knowledge workers care about doing a good job, but they also don’t like to put in effort. They choose their effort and the way they want to do the task to solve this problem:

The expectation is with respect to the knowledge worker’s subjective knowledge of the best way to do things. Because the expectation minimizes the square loss, the best z will always be m. By definition, E[(s-m)²] = v. Because this problem is convex for positive e, we can solve for the effort level from the first order conditions:

So, at a high level, uncertainty defines effort. The more uncertain we are, the more effort humans supply to guard against unexpectedly large errors in the course of action they decide to take.

Therefore, the (objective) time it takes a given worker to do a task without an LLM is:

LLMs as pure automation machines

LLMs work by receiving context from humans. Humans can transmit their beliefs to LLMs as context, and then the LLM decides what to do with it, i.e., the LLM production technology is:

Where a gives the equivalent effort level of the LLM’s output. It measures the LLM’s accuracy or performance and might be captured directly by how long it takes to coach the LLM on exactly what you want it to do or on the quality of the output it produces.

Our first model of LLM production is what I’ll call “the pure automation model.” In this model, the LLM performs the task in the same way a knowledge worker would, but effortlessly. It doesn’t have its own beliefs. You tell it what you want done, and it does it. So: f(m,v) = m, because you’ll choose to tell it to minimize the expected time given your beliefs, and the (objective) time it takes to complete the task with an LLM is:

It’s a simple model, but it actually has some semi-non-obvious predictions.

Workers choose to use LLMs when they are MORE certain of the best way to do things

Workers choose to use LLMs-as-automation when they are more certain about how to do things.

The result follows from effort substituting for uncertainty. Workers can put in more effort to mitigate their uncertainty when using the non-LLM technology, but the LLM technology doesn’t respond to their effort.

So, workers choose to use LLMs-as-automation whenever their uncertainty about the best way to do things is low enough:

So, with the introduction of LLMs, only the more ambiguous tasks will be done by the effort technology, and, as the effectiveness of LLMs (a) increases, more and more ambiguous tasks will be done via LLM.

The time to complete tasks increases for some workers when they have the LLM option

This result is “obvious” when you write down a model like this, but not clear at all without actually writing down the math. The key idea is simple, though:

Workers want to avoid spending effort, so they will choose to use LLMs even if the effort technology is faster than the LLM technology.

We already have the rule for when workers will choose to use LLMs. So, to characterize when LLMs increase the time a task takes, we need to look at the conditions under which LLMs are slower:

So, for v = 2a³, the LLM method of production is slower, but workers choose to use it (because 2a³ < 8a³); otherwise, they have to pay the effort cost. Folks don’t just try to minimize time (equivalently, maximize output quality).

For a moderate level of uncertainty, i.e., for a³ < v < 8a³, knowledge workers choose to use LLMs despite the longer task times (equivalently, lower-quality output) they entail.

This production model is static, but it implies something about dynamics. If AI quality (a) improves only slowly, some industries stall in this middle ground, where workers choose to use the LLM technology, but it gives worse/less output. Firms using the effort technology outcompete them. So, AI adoption is quick in industries defined by moderate ambiguity about how best to do things for firms that make it readily available to their workforce. AI adopters lose out to the Reactionaries: firms that force workers to use the effort technology and refuse to pay Anthropic or OpenAI.

Automation LLMs decrease the returns to accuracy if uncertainty is low

Do more skilled workers do better or worse with the option to use LLMs? Do LLMs compress the productivity distribution, allowing less-skilled (maybe early-career) workers to catch up to more skilled workers—or do they widen the gap between the skilled and the less-skilled?

Suppose everyone has the same level of uncertainty (the same v). Then we can define more skilled workers as those with a smaller d = (m - s)².

Shrinking d decreases time by v^(-1/3) for the effort technology and by a^-1 for the LLM technology. So the reduction in time from more accurate prediction is greater for LLMs if:

This is the same as the rule for when LLMs are slower. So, for highly uncertain tasks, the return to skill is greater for the LLM technology than for the effort technology. For low-uncertainty tasks, the return to skill is greater for the effort technology.

If we incorporate the choice constraint into this, we can partition the results like so:

Without LLMs, the return to skill would always be v^-1/3. LLMs increase the return to skill for moderate uncertainty but do not affect it for high-uncertainty tasks (because workers choose effort over LLMs). For low-uncertainty tasks, LLMs are faster than the effort technology and reduce the return to skill.

So, moderate-ambiguity tasks will demand more skilled workers with LLM than they did with the effort technology. Less ambiguous tasks will demand less skilled workers.

So…

If we can compensate for our uncertainty about how best to do something by increasing effort, then we will only do tasks with low uncertainty via the LLM technology. Skill will matter less for low uncertainty tasks. And tasks with moderate uncertainty will now take longer (or be lower quality) than before (tell me you haven’t felt this at some point at work since LLMs came into their own) while demanding higher-skilled workers because of the increased return to skill.

In this model of LLM production, the knowledge worker provides a vital input: their own beliefs about how best to do things. LLMs don’t work without the worker. The model helps us isolate the effects of the automation component of LLMs, but, of course, LLMs do more than automate production (although for some applications… that is pretty close to their purpose).

In the next model of LLM production, we’ll allow the LLM to have beliefs of its own about how best to do things and open up the decision of whether to use a worker at all…

Math…

Although this model of production is relatively simple, it makes arguments that get bandied about in the AI discourse precise and concrete, with at least a plausible theory for what gets done by AI and what doesn’t. Writing down exactly what you are assuming about the structure of folks’ decision-making process clarifies. Implications naturally follow from assumptions and structure.

You can do more than just say things.

Saying, “I think LLMs are great for generating quick queries or searching customer calls, but they’ve made docs less useful because they’re all AI-generated and very hard to parse,” is all well and good, but it’s just like your opinion, man. With a clear model, it starts to make sense.

Writing a good doc is a great example of a moderately uncertain task. It’s not untrod ground, but neither is it exactly obvious how to propose a new idea effectively in a way that accurately communicates what you want to do, why it’s a good idea, and what might go wrong. So, the degradation in quality with the LLM technology available makes sense. It follows from folks avoiding the effort cost to work on a moderately ambiguous problem.

On the other hand, a quick query or labeling customer calls is something you know exactly how to do. It would just take you a relatively long time to do it yourself. And the model says that low-uncertainty scenarios will be better with AI because you can get a baseline effort level without workers having to pay the effort cost for the exact kind of task where the returns to effort (from the worker’s perspective) are low.

The advantage of writing things down precisely in math language is that it makes it clear what assumptions drive your thinking, and it creates natural implications of those assumptions that you’ve not thought of before.

Writing, in both Human Language and Mathematics, is how humans think. Computers have RAM. We’ve got Emacs, Google Docs, 3x5 notebooks, and cave walls.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

We’ve got a few stories from the SaaS wars—and takes, of course. We have an infinite supply of takes, far exceeding demand.

Subscribe now

• Dispatch from the SaaS data access wars:
  Every week, there’s a new story about so-and-so blocking data access from such-and-such. This week, Microsoft is supposedly blocking a certain kind of access from external tools to Power BI. The article contains this haunting quote: “The friction centers on Power BI, a Microsoft product nearly all Fortune 500 firms use to analyze data about their operations in charts and other visual formats.” Really? Nearly all?
  
  I’d like to take this chance to share a conspiracy theory. Power BI isn’t real. I hear about it all the time. It’s allegedly everywhere. And yet… I’ve never used it or seen it be used. I’m tech-old. I’ve been around lots of data environments. Sounds fake to me.

• Did you hear SpaceX is going to be IPO?
  No? I didn’t either. I couldn’t find a single article about it to share. See if you can find one somewhere on the internet.

• Customers demand shorter contracts over uncertainty of “what comes next”
  There’s a story on AI dollars crowding out traditional business software purchases. The pool of dollars isn’t fixed, and if there is a large return to these AI products, why couldn’t these firms borrow to cover token purchases and repay the loans with excess profits? So, something is wrong here. Either credit markets (unlikely), the returns from AI spend are currently negative (plausible until companies learn the right way to deploy it at their firm), or there’s a principal-agent problem (highly likely). The person doing the buying has a fixed budget, so even though the company could just expand borrowing, there’s no cheap way for the agent to pass the information up the chain.
  
  The second half of the story is more plausible, but speculative. Companies are worried that a wave of new, better products will emerge over the next few years and are reluctant to be locked into a long-term contract. That makes sense, but I really haven’t seen any evidence of useful AI-native business software, aside from the models and coding stuff coming from the primary labs. The AI components added to existing SaaS have been useful, but AI features are not really useful on their own. You need something that works well for all the more mundane features of the app first, and that turns out to be a hard problem.
  
  It reminds me of every time I’ve seen what I think is an opportunity to exploit in an industry. I realize that, while I might be right about the opportunity, to actually profit from it, you’d have to launch a competitor that does all the other aspects of the industry right and then just adds the new insight. That’s very hard to do.

• Intuit moves to consumption-pricing.
  Everyone is doing it because it’s the only thing you can do, now that the Variable Cost Revolution is well underway. What is the Variable Cost Revolution, you ask? For a long time, one reason software companies were so profitable was that their costs were almost entirely fixed. Costs did not increase very much as they added more customers. So, with scale, they made oodles of money. But AI has changed that. If you run an AI feature in your software, you have a variable cost product. As demand for the product increases, your costs increase meaningfully. LLM-based features have changed the cost structure of the entire industry, and the pricing structure follows the cost structure.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve been thinking about the fact that statistical modeling doesn’t really have a standard declarative language for specifying a model—and how there’s nothing really preventing that from being so. I have this hope that were such a language to exist, people would finally understand the difference between an estimator and an estimand.

Subscribe now

I started thinking about this recently when I wrote a little something for a side project—coming soon—in STAN.

[I style the name in all-caps for unknown reasons. It’s not an acronym (I think).]

If you’ve never used it before, STAN is a domain-specific language for likelihood models, primarily for Bayesian inference. The core idea is that you can specify the model in this declarative syntax, and the STAN program computes derivatives and the path for Markov Chain Monte Carlo iterations of the posterior distribution using automatic differentiation in a fairly well-optimized way (so, it’s nearly as fast as this sort of computationally-intensive thing can be). It’s super nifty.

To see what it looks like, here is an example of a STAN specification from a model I wrote for an old project.

[If you’re curious: the “old project” is an app that creates an “optimal” schedule for working towards a long-term goal based on your current, observed work behavior (i.e., you like to work on Tuesdays, you like to work in big chunks, you like to frontload or backload work relative to deadlines, etc). You don’t enter your preferences. The app learns from your actual work behavior to help converge on an “ideal” work schedule across projects over time. It never really got any traction, but I liked the idea.]

data {
  int<lower=0> N; // number of days so far
  int<lower=N+1> T; // when the work has to be done
  array[N] real logit_remaining_share; // logit{W / (Y-work prior to W)}
  array[T] int G;
}

parameters {
  vector[6] dow;
  real<lower=0.0> sd;
  real weekday;
  real<lower=0.0,upper=1.0> r;
  real<lower=-0.20,upper=0.20> b;
}

transformed parameters {
  vector[7] K;
  for (i in 1:7) {
    if (i <= 5) {
      K[i] = exp(dow[i] + weekday);
    }
    else {
      if (i==7) {
        K[i] = 1;
      }
      else {
        K[i] = exp(dow[i]);
      }
    }
  }
}

model {
  vector[N] future;
  vector[N] present;
  for (i in 1:6) {
    dow[i] ~ normal(0, 1);
  }
  weekday ~ normal(0, 1);
  r ~ beta(1,1);
  sd ~ gamma(1,1);
  b ~ uniform(-0.20,0.20);

  for (i in 1:N) {
    future[i] = 0.0;
    for (j in (i+1):T) {
      future[i] = future[i] + ((1+sqrt(j-i))^b)*(K[G[j]]^(1/r));
    }
    present[i] = K[G[i]]^(1/r);
  }

  for (i in 1:N) {
    logit_remaining_share[i] ~ normal(log(present[i])-log(future[i]), sd);
  }
}

The key idea here is that you can just describe the prior distributions [weekday ~ normal(0,1)] and the likelihoods [logit_remaining_share[i] ~ normal(log(present[i]) - log(future[i]), sd)] and then STAN simulates the posterior distribution using a fancy Monte Carlo. This makes it easy to specify flexible, realistic priors and likelihoods without relying on computational tricks (like conjugate priors). So you don’t specify models based on computational concerns, but on how well they describe the data-generating process. Plus—and I like this part a lot—the model’s description is fully separate from the details of how it’s estimated.

[Yes, I frequently write code without descriptive variable names. I think this is the good and proper way to write math code when you have a model you’ve written out somewhere in math, i.e., the names here match the LaTex, which is the source of truth, and the real documentation. But your mileage—and ability to get such things past engineers reviewing your PR’s—may vary.]

But I don’t do many Bayesian projects. I mostly work firmly in frequentist land. I don’t do much likelihood-based estimation either. Broadly, I use moment-based estimators, but I really like the idea of a domain-specific language for modeling that’s portable across R, Python, Fortran, Julia, whatever.

I’ve been sketching out what this would look like (I think the cool kids call this “building in public”).

Here’s my current draft:

data
  N;
  K;
  J;
  X[N,K];
  Y[N];
  Z[N,J];
  T[N];
end

parameters
  b[K];
  te[1];
end

latent
  # parameters can show up here
  Y{T} := start |T, t|
       Y if T = t else unobserved;
  end
end

assumptions
  # no parameters or latent variables can appear on the right side of the := operator
  moment(Z @ (Y - X @ b)) := 0; # iv assumption to get b

  moment(te - (Y{1} - Y{0})) := 0; # define treatment effect
  moment(Y{T}) - moment(Y | T) := 0; # exogenous treatment assignment
  
  # alternatively, define some CIA-like assumption
  moment(Y{T} | X) := moment(Y | T, X)

end

Another example for a discrete choice demand model:

data
  J; # number of products
  K; # number of characteristics
  N; # number of customers
  X[J,K]; # product features
  B[N,J]; # dummies for whether each customer bought
end

parameters
  b[K]; # characteristic parameters
  a[J]; # product fixed effect
  choice_prob[N,J]; # customer probability of choosing each option
end

assumptions
  do i from 1 to N
    do j from 1 to J
      choice_prob[i,j] - 1/(1+exp(-1*cross(X[j,:], b) - a[j])) := 0;
    end
  end
  do j from 1 to J
    moment(choice_prob[:,j] - B[:,j]) := 0
  end
  moment((choice_prob - B) @ X) := 0
end

A key point here is that “moment(…)” is not just a function being applied to an input. It’s part of the language, and the language knows the properties of expectations. So it can, for example, know that:

• E[Y{1} | T = 1, X] := E[Y{1} | X] => E[Y{1}] = E[E[Y{1}|T=1, X]]

You can also centralize moments with “central_moment(…)” and get higher order moments by using “moment(X, 2)” or “central_moment(X,2)” (i.e. the variance).

This isn’t a project I’ve written code for yet. I’m still trying to decide whether this makes sense and is useful enough to spend a decent chunk of time on, but in principle, I think having a declarative syntax for moment-based modeling makes sense.

The core, workhorse estimator behind it all would be the Generalized Method of Moments, because it’s flexible enough to cover all of these cases. Maybe it can even include inequality restrictions…

Anywho, thank you for listening to me ramble about ideas I’ve been bouncing around to improve statistical programming. I really like the idea of more firmly decoupling the estimand from the estimator in the code itself, so I’ll probably take a shot at making this a reality once I’ve thought about the problem more precisely and have a good grasp of how to implement it right.

What do you think are the biggest gaps we have in statistical programming?

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I would have liked prediction markets if I’d thought about them much before I saw them in the wild.

I’m a fan of markets. Why not learn what people really think by having them put money on the line? I do like forcing pundits to bet. We should tax bullshit. It has negative externalities.

But now, we have data, and, unfortunately, we have to use it.

Subscribe now

Another argument I would have found convincing in the past: who cares if there’s insider trading? In fact, that will just make the markets better at predicting because all information, including that available to insiders, gets incorporated into the price.

The idea that information is valuable is central to economics and intuitive. The more information is reflected in prices, the better decisions we can make about capital allocation. The misallocation of capital is a major drag on productivity.

A few things were missing from my model of the world:

1. The value of information is heterogeneous.

2. The value of information does not correlate with market size.

3. Not all information is disperse and diverse, which is where markets shine.

4. There are, naturally, negative externalities to gambling. So, the market's social value does not start at 0.

For example, one of the larger prediction markets was a bet on who would win the New York City mayoral race. What is the value of this information before the election concludes? Perhaps it provides a means to hedge against a certain candidate’s effect on asset prices if he wins? Or, aggregate information from this market’s outcome can be used to trade in the asset market.

Markets do not create information. They reveal and aggregate the information people bring to them. So, what additional information is the market aggregating here? It can’t come from public polls, which are, well, already public. It does generate a numerical index of the “vibes,” which are public knowledge but difficult to quantify. It’s not clear how predictive the vibes are, though, and, in any case, they are already reflected directly in asset prices. From insiders, it incorporates private polling data, which is probably the most useful information generated by the market.

But couldn’t those insiders use that private polling data to directly trade on assets that will (allegedly) be affected by a certain candidate’s win? Then, that information would already be in asset prices.

No, is the answer, I think, but the reason for that is that political insiders are small in asset markets, so they can’t really move the price, but large in (some) prediction markets, given their current size.

[I have an armchair theory that media fascination with prediction markets is that polls report their results in terms of proportions, while prediction markets return results that give an implied probability of so-and-so winning. Of course, polls could also return probabilities of winning with some kind of Bayesian inference, but they don’t. An opportunity for a polling company.]

And therein lies the reality of the information value of prediction markets: it depends on people with meaningful information making up a sizeable enough percentage of the market that their private information gets reflected in prices. If the only people with meaningful information are insiders and insiders are small, then they earn profits by trading in the market because their bets don’t move the price, but we don’t learn any special information from the market that we don’t already know.

The prediction markets that would be most useful are not political horse races or sports gambling, but predictions about events where the information is difficult to quantify and aggregate into a belief in the abstract, and where information is diverse and widely distributed.

For example, the market for apples aggregates information on the value of apples. Everyone has their own apple preferences, so the information is very diverse. No one else knows your preferences, so the information is widely distributed. Markets aggregate all this diverse and dispersed information into a price for apples, which summarizes how apples trade off versus any number of other goods.

Markets like these are not popular topics on Kalshi. Look at the home page. I struggled to find topics where many individual people have meaningful, diverse information about them. I’ll discuss a few that were (arguably) close to that before jumping into what’s mostly there.

• “Will Americans receive stimulus checks?” Of course: how would any non-insider have useful information to contribute to the market here? But, it does, I suppose, operate as a hedge of some kind in case the government has to borrow to write stimulus checks, and so it might offer some idea of how exposed people are to the demand for treasuries or something (which is diverse, dispersed information)… but there are already so many ways to do this that have a market size of more than $1.5M.

• And, sure, there are some markets here that are more or less just typical financial instruments. There’s one directly betting on when Bitcoin will cross $100k again, which is just a typical financial market that aggregates individual demand information. But, again, this isn’t and wasn’t really the pitch for prediction markets. This is just a normal financial market.

• “More tech layoffs in 2026 than 2025?” While there’s some alpha from being an insider, it’s limited by the fact that you only know your company, and the market can effectively aggregate all these signals about the tech industry more generally. Workers without direct knowledge of pending layoffs do have some information on them. You can usually tell when it’s coming, and such a market might be a hedge against the possibility you’re laid off. Others outside the tech companies might have some useful leading indicators worth aggregating as well, e.g., that it’s getting harder to close commercial real estate deals in the Bay Area.

The combined market size for these and a few other roughly information aggregative markets is dwarfed, of course, by the $293M market for “PGA Championship Winner,” which is what these sites are really about.

“For where your treasure is, there your heart will be also.”

If you write down a typical model for a prediction market, you’ll start saying things like “each individual gets a signal x…” but they don’t really. Not in most of these. As much as I’d like to believe I have some signal about what’s going to happen with geopolitics, I don’t. My beliefs on the likelihood of various outcomes of the war in Iran are just an index of public information. I have no new signal to offer the market.

So, then, what’s left? If we don’t really draw signals, then we’re left with non-insiders really just placing bets on the events because they are risk-loving, either because they are in a relatively flat part of the wealth utility space and need to get a great outcome to move up a level, or because they have “an innate preference for risk.”

Go to Kalshi and ask yourself: what price in each of these markets would cause me to enter? Most of these, within a very broad, reasonable range, I would have no reason to think are mispriced, even if the price were like +/- 20% from what it is currently. I’d just assume the market had more information than I did.

The same isn’t true in the stock market. For example, if the price of Apple were 20% lower than it actually is, I could look at their financials, public roadmap, and go, “Wait a second. This is wrong. I am going to make a boatload of money. Can I take out a second mortgage on the house?”

So, this is why I think prediction markets have failed in reaching their promise:

1. For most of the predictions we want the markets to make, relevant information is highly concentrated.

2. There are very good substitutes for the more meaningful markets that directly focus on more traditional financial transactions and don’t rely on people with “innate preferences for risk” to get liquidity.

I’m not sure that we should like a system that functions as a transfer payment to insiders from people who think the key to improving their lives is to put it all on black.

I’m not for banning prediction markets, of course. People should generally be free to do as they will. Many vices and antisocial behaviors are legal.

We still let people be Dodgers fans, among other things.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

This was a strong week for pricing news, so I’m leaning into it with a round of a few choice pricing/monetization/data/statistics stories from the week. Maybe it’ll be a recurring thing (actually recurring, not “recurring” as in ARR).

Subscribe now

1. Some Anthropic customers don’t like its pricing. I heard not so long ago that AI was ending scarcity and replacing white collar work by 2030 at the latest. Now, even the Silicon Gods have a price, and some folks think it is too much. Just imagine if the robots cost more than our glorified address book! That’s the contradiction of LLMs. They’re fantastic products—amazing, in fact, but if they were as good as the AI labs’ frontmen say… they’d cost more. They’re more than worth it at current prices, of course. “Many technology firms and other large Anthropic customers say they plan to eat the soaring costs...” Price competition is tough because of relatively low product differentiation between the top players, which should keep prices down… but even Google can’t light money on fire forever...

2. Speaking of the cool, currently cheap-enough AI… I haven’t had the chance to play around with it yet, but the demos of Hex’s new “generative apps” look snazzy. If you don’t know Hex, it’s a notebook ala Jupyter that makes it easy to spin up web apps powered by the notebook for your stakeholders’ consumption. It’s well-done, and the stylings of these new vibes-powered apps look fun.

3. Spotify is running this hilarious monetization play. If you open Spotify in the browser and switch tabs, the miniplayer pops up in the lower-right. If you are not on Spotify Premium, like me, it shows the miniplayer covered with a prompt reading, “You discovered a Premium feature. With Spotify Premium, you can make the miniplayer even smaller.” I had never noticed the size of the miniplayer before. It had never occurred to me that it was too big, but after I clicked “Not Now” and the player ballooned to its previously normal size, I saw how large it truly was. Like Adam in the Garden, I had tasted the fruit of a properly-sized miniplayer, and lost my innocence. I felt a sense of loss of what was once and what could be again… if only I didn’t refuse to pay $120/yr or whatever Spotify Premium is. I don’t mind ads. It’s nice to know what the fellow corporates are up to.
   
   Seriously, a very clever surface to drive conversion on. I love it. I bet it works.

   

4. Slowly but surely, AI app pricing is matching the AI cost model. In the long run, cost structure sets pricing. Take Figma, for instance. They started capping how much customers can chatter with the Clankers for a flat rate, like the old cell phone plans. Now, you buy credits on top of the seat price once you cross a certain usage threshold. This always had to be the case, eventually. It’s not just Figma, of course.

   

5. These rectangles sold for $85.8 million, so next time, Just Monetize It (tm). Someone will buy it.

   

The Information owes me a referral fee for dominating the links, but they had great stories this week. Nice job, folks!

Blog and Side Projects Updates

• Crossed 100 subscribers this week. Super-Neat! Hope you enjoy reading!

• Completed our first full week of the new, thrice-weekly posting schedule! I want to write well, and it turns out you have to practice. Go figure.

• We made a small chunk of progress on the last main project area for my secret side project (a webapp for this Common Statistical Task). We got to write some Stan for the first time in a while.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve complained about the demonization of pricing experiments before, but alas, there’s no arguing religion. Sometimes, you just have to accept that you can’t run price experiments, and still, They want you to estimate demand.

The trick, of course, is to convince Them to let you randomly send some folks a 10% off coupon for whatever you’re hawking. Often, the marketing/promo team has already done this, so you can just steal their data. This is best.

The idea here is that the promo introduces price variation, but you have to be careful with how you use the data to identify demand. So, thought I’d write a little post on how to do it. Price well!

Subscribe now

A simplified empirical demand model (ignoring all the nonsense you control for, etc) looks something like this:

Where j indexes products and t indexes time.

The problem is that historical price changes are related to all the other factors that affect output (E). For example, often this equation more or less describes the price variable:

i.e. the price increased at some point in time in the past. It is extremely unlikely that nothing else changed with demand before and after the price change. Demand is always changing.

So, you can’t just run the regression.

The standard solution to this problem is to use an instrument that is uncorrelated with the other factors that drive output (e.g., demand shocks), but is correlated with price, because then:

So, you can just solve for the elasticity β.

But you don’t have any instruments. You know this. I know this. The supply of instruments is much lower than the demand, and, if you’re working in industry, it’s just extremely unlikely that some natural experiment exists for your particular company.

So, you have to make your own instruments. That’s what experiments do. Experiments are instrument factories. They make random variables that are uncorrelated with everything but what comes after being bucketed into the experiment.

But the 10% off coupon mentioned earlier isn’t a valid instrument for this demand model, sadly.

It’s randomized across users, not products. So, there’s no reason to suspect it’s excluded from a direct shock to demand. The randomization ensures that variant assignment at the individual level is uncorrelated with other user characteristics, but this doesn’t solve the endogeneity problem, which is cross-product and time!

The problem is that this demand model is based on product-time variation. So, to use this instrument, we’ll need to build up a customer-level demand model and derive the aggregate demand for each product from the individual demand curves.

A natural way to do this is with a discrete choice model. To save on notation, let’s suppose you’re hawking one product, so that this amounts to modeling the conversion rate. Here’s a pretty standard model, but you can use something more flexible:

Where X_i are some customer characteristics and P_i is the final price of the product for customer i. Here, we’re allowing for other reasons a particular customer might have a lower price aside from access to the 10% discount. P_i is just whatever the customer’s final price ends up at.

Now, let Z_i = 1 if the customer gets the 10% promo and 0 otherwise. The key idea is to use this Z_i as an instrument for price.

For doing binary choice models with instrument variables, I usually use a little something I wrote circa 2017-2018: repeated two-stage least squares.

The basic idea is that you can show that doing the following procedure is a contraction mapping:

1. Run two-stage least squares for the linear probability model of 1(purchase_i) on (X, P) with (X, Z) as instruments. Call this estimate c.

2. Set k = 1.

3. For some initial guess of a and b, compute the predicted conversion rate. Run the same two-stage least squares estimation with the dependent variable being the predicted conversion rate. Call this estimate c_k.

4. Update: [a_k+1, b_k+1] = [a_k, b_k] + c - c_k.

Under an assumption that the instrument has a “strong enough” relationship to (X,P), this procedure converges to the true a and b in a large enough sample. It’s also got standard normal, asymptotic inference because there’s an analogous GMM estimator.

The key thing is that you can use these kinds of experiments to generate plausible instruments for demand estimation even when you don’t have the ability to do more direct price experimentation (for whatever reason).

Once you have this model, you can simulate changes in price by just plugging in new prices, but identification here has its limits. All you’ve done is exogenously moved prices 10% for some customers. If you start looking at 30% price increases, you’re inferences will be based more on functional form than anything in the data. The real info you want to get is “Are we over/under-priced? Would a small increase/decrease help us?”

A couple things to try to improve power and flexibility here:

1. If you can get more promo depths (lower or higher discount %), that’s super useful because it will give you more instruments. Maybe you can get a more complex model of the pricing curve. With only the one 10% promo, we can’t introduce a P² term into the model because we don’t have a separate instrument for it. But if we have a few different treatments, we can get it in there.

2. If you can get them to send out the promo a few times where you re-randomize the recipients, this is good, because it generates additional variation in variant assignment, giving you more power.

There’s an important wrinkle here. The e-mail or notification that tells a customer they have a promo itself might be responsible for increased demand, i.e. aside from any discount the customer gets.

The trick to deal with this is to send promos to everyone but vary the discount level. Instead of doing 10% vs 0%, do 20% vs 10%. Then, you can send the same e-mail to everyone and just change the discount. This helps isolate just the pricing impact of the experiment.

At this point, we might ask: “Well, if we’ve got this nifty trick to run a more palatable experiment, why would we ever want to run pricing experiments?” This is another case where it’s important to remember that the data’s only really informative in a neighborhood of actual prices people are seeing. This strategy only tests prices below the status quo, not higher. So, you’re not really getting direct evidence on what would happen at higher prices. You’re just assuming that the elasticities you see for the 10% lower price are roughly relevant when the price is 10% higher. That’s better than vibes and purely observational analyses, but it’s an assumption all the same. So, you have to be more cautious when increasing prices than decreasing them using this method.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve become less skeptical of surveys. Economics, as a field, has a bias for revealed preferences—what people do—over what they say, and I mostly inherited the biases of the field because, by default, you believe whatever you read when you were twenty-three.

But I’m becoming a fan of surveys.

This post is about both sides of the revealed vs. stated-preference divide. I talk about the problems with using surveys to package and price products and ways to replace some common survey use cases with revealed preference methods. But also about why sometimes it pays to just ask people what they’d do, and believe them.

Subscribe now

The difference between saying “I’d pay $100” and writing a check for $100 is the difference between the plan to run a marathon and lacing up. This is the fundamental problem with surveys.

Conjoints don’t solve this problem. Artificially creating a realistic choice set doesn’t bridge the gap between stating and paying.

[My little take here is that for the narrow task of pricing (as opposed to understanding the determinants of demand), conjoints are worse than just asking “would you pay $5?” My evidence is classified, but I’ve seen many price elasticities from conjoints, and they are always wrong to the point of being useless, while I’ve seen decent-enough results from the “would you pay $X for ___?” question.]

The right argument for running a survey is that we don’t have a good answer to the question: “what else could we do?” When there’s nothing else to do, you run a survey. But there’s a temptation to throw in the towel early because (we imagine) a survey is a time-bound, well-defined task that we can probably ship in a week or two.

But you can learn a lot from revealed preferences before you have to resort to stated preferences.

It’s rare to launch a genuinely new product, a product that’s completely unrelated to whatever else your company does, so you don’t have any relevant data. Most new products are re-packaging your existing product or a small new add-on. You want to offer an all-you-can-eat subscription, alongside your current à la carte, transactional business. You’re going to start charging for a type of product usage that was previously free. Etcetera.

When revealed preference data exists that answers the relevant question, it’s better because it represents real checks. For example, it might look like you don’t have any data on how people will respond to a price for a previously free product, but you can triangulate.

• When you introduced this product, how many more people signed up for the subscription?

• When you raised the price by $5 last year, how many fewer people did?

Combining those two answers tells you something about how valuable customers find the feature. You can say things like “people value the feature like a $3 reduction in the product’s price, so…” [You do need both parts here: variation in the presence of the feature and variation in the price.]

The point is that, if you’re a wee bit creative, you can use data to answer a broad class of questions before you need to make your own data via surveys.

Surveys are worse than behavioral data, but they are 1000x better than vibes (the 1000x, however, was measured by vibes).

Analysis is a competitive market. If there’s no data analysis or survey, decisions will be made via vibes. There’s always an outside option. People don’t have to use any evidence at all to make decisions. You can just decide things. Vibes are always a backup plan.

[This is why I lean towards “give stakeholders something” instead of “we don’t really have any ready data on that” or “oh, that’ll be involved, we might be able to work that into the next sprint” (at which point the data is, to a rough approximation, always useless: the decision’s been made). The question is: “Do you have something that’s probably better than vibes?” If so, ship it.]

The point of a survey is to create data where we don’t have it. The typical types of missing data are:

1. We don’t have price variation. The price has never changed before.

2. We don’t have product variation. Either everyone has always had the product, or no one has ever had it. So we don’t know how much anyone will like it.

3. We have no way to connect the new product to an existing product (it’s not just a re-packaging of goods we’re already selling).

It’s surprising how many problems in statistics amount to missing data problems. Causal inference is all about the fact that we don’t observe the potential outcome for treatments that people never experienced. Pricing new products is the same kind of problem. We don’t know what people will pay for products they’ve never been offered.

This is the key frame to use when writing surveys: what data is missing? We don’t just want to run survey to run a survey. We have a goal. We’re trying to fill in the gaps in our data. Answering this question makes writing a focused, short survey easy and natural. We need to ask questions that extract the missing data.

We don’t have price variation? Then we need to ask people, “Would you buy it at price X?” and “Would you buy it at price Y?” Bam. Now the price varies between X and Y.

We don’t have product variation? Suppose we’re looking at adding a new feature to a bundled subscription. Then, we need to ask the questions “Would you buy it at price X if it included feature Z?” and “Would you buy it at price X if it didn’t?” i.e., we need to keep the pricing constant and vary the feature set.

Another kind of missing data we need to extract from a pricing survey: the respondent's size. If you’re selling, say, a B2B product, the size of the company purchasing the product will be relevant to how much this customer’s price sensitivity matters. If you’re doing seat-based pricing, then you need to know the number of seats the respondent represents. If you’re selling a transaction-based or usage-based consumer product, you have to get a sense of how many potential transactions or how much potential usage the respondent represents. The particulars of how to do this depend on how demand scales for the product, but it’ll usually be something like “how many times do you run workflow X” or “how many employees work at your company”.

From there, you can maximize revenue/profit (depending on your cost structure) weighted by the respondent’s demand size (a lot of survey tools don’t do this sadly: they’ll just maximize revenue for you without taking into account demand size—alas). The demand curve is just given to you by the highest prices that people say they’re willing to accept.

I like to solve a problem that looks like this:

Where P is the price, W(i) is the weight associated with the size of the respondent’s demand (maybe based on their answer to company size questions, etc), WTP(i) is the highest price folks say they’re willing to pay (the survey usually gives an upper and lower bound on WTP, so it makes sense to solve this at each of those extremes), a is a discount or premium on people’s stated willingness to pay. I like to check for robustness to deviations between stated willingness to pay and true willingness to pay, or even just to downweight WTP if you’re trying to price for growth. So you can plot the optimal price as a function of a and find a price that doesn’t vary a lot for small deviations in a.

There’s a bit of an art to this.

For a well-targeted survey, a very high percentage of responses to the “Any other thoughts?” prompt are really thoughtful. This isn’t a real quote, but it’s illustrative of responses you’ll get: “Seat-based pricing wouldn’t really track the value here. We’d really only need one seat because only one engineer needs to manage this. The usage option you presented makes more sense, but only if there’s some cap or it gets cheaper as we scale…”

Humans like being helpful, and even though it takes them time, and it doesn’t really affect whether they get the $25 gift card, respondents take surveys seriously. Because humans are generally good folks who like to help their fellow corporate drones out. Go figure. The kids call this a “white pill” (I think?).

It’s not like you have to blindly roll out the survey’s optimal price. A couple of in-depth interviews with potential customers can at least make sure you’re not going off the deep end. If someone says, “That sounds reasonable,” it probably is. If they say, “lol what?” something’s probably gone wrong.

You don’t have to completely discard vibes. Vibes contain useful data. They’re a statistic of past experience. So, you can use them to discipline the survey, while still basing pricing (or other decisions) on data. If some respondents’ stated willingness to pay makes no economic sense, just drop them. If you’re worried people are overstating their willingness to pay—before they have to actually convince Finance to sign off on it—then just underprice it (set a < 1 in the above problem).

The point is that you can build trust in survey results. You don’t have to treat them as a machine that takes responses and converts them to decisions. This makes them infinitely more usable, despite their limitations.

I’ve been thinking a lot about surveys recently. There might be a few more survey-focused techniques in upcoming blogs (thinking through some issues).

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve decided to write well.

Now, I know the first step traditionally is to move to Paris or a small American town—you can get away with NYC if you’ve got old money—and preferably not be employed in B2B SaaS, but I don’t have that kind of dedication. I like money, I have suburban tastes, and I like my neighbors like I like my comments—anonymous.

So, I’m going to try another tactic: practice.

Subscribe now

I’ve been blogging once or twice a month on statistics (which you call “data science” when you’re trying to get a job) for a few years now, but I’m going to write more frequently now. My goal is to put out a blog three out of every seven days.

Of course, that doesn’t mean I’ll write up three experimentation techniques every week, like in the old blogs. I have a limited supply of these! But I realized I had a lot of thoughts about industry tools, practices, news, etc., and just listing the ideas in my little Kanban board made it clear there was enough content to do it.

So, we’re broadening our horizons, limited by the following principles:

1. I have to care about the topic.

2. I have to know enough about it to not sound stupid (hard!).

Especially looking forward to doing more tools-of-the-trade reviews! I did one recently on ggsql. There’s a ton of products, tools, techniques, etc that I’ve never really used because they weren’t used at prior jobs—or, they’re new—and it’d be fun to take a look at them.

I’ve got some initial ideas, but feel free to pitch me on tools to review! Including ones you wrote! (With the warning that I will say what I think of it. My rate for lying about how great your tool is $1 million. So readers should know that either I think the tool is really good or you paid me $1 million.)

Looking forward to the challenge! My current plan is to write the posts mainly over the weekend and schedule them throughout the week, but we’ll see how it evolves…

Zach

Connect at: https://linkedin.com/in/zlflynn

Posit released a new library for graphics programming (plotting) in a SQL-ish style language called ggsql. It’s aiming to be like its ggplot project for R/Python, but with a SQL-ish syntax, which, I guess, is becoming more and more the language of everything (remember the NoSQL moment in the early 2010’s… yeah, that’s dead).

Subscribe now

I gave it a spin over the weekend on some “ancient” data I re-discovered in an old Dropbox folder. In grad school, I scraped the internet for this data on radio station coverage and formed markets from sequences of intersecting stations for this paper I was working on.

The site was trying to stop scrapers, so I had a Perl script randomly cycle through a set of User Agents and switch Tor proxies frequently while downloading thousands of pages (it blocked every 1-5 and the algorithm tried to find the right combo of waiting + user agent switching + tor proxy restart that would unblock me). The site’s solution for stopping scraping was pretty good. It took my little eeePC netbook (remember those!) weeks to get all the data with all the switching, random pausing, and delays I threw in there to get past the detection automatically. I just kept the laptop open night and day. The big hole in their scraping-prevention technique was that they assumed the scraper valued their time and sanity more than paying them $100 (I think it was about that much to get a computer-readable form of the data). Their model of the world did not include a graduate student making ~$10k a year, who was perfectly content to parse HTML and wait weeks to save $100.

In any case, I haven’t looked at this data in forever, so it felt like the perfect laboratory to poke around with ggsql.

Funny coincidence: this paper was the last project I worked on, where I used lattice for the graphs, not ggplot.

What ggsql does (with a few examples)

ggsql uses a “reader” that processes the pure SQL parts of the statement and the ggsql extensions. The easiest way for me to test things was with the DuckDBReader, which lets me visualize the output of DuckDB queries. I imagine there are other readers for different databases (or, there soon will be).

ggsql outputs Vega (a JSON format for graphics). Vega makes a lot of sense as an output target because it’s flexible, low-level, and plays nicely on the web side + it’s got some institutional support, and tools on top of this are sort of the key to making ggsql valuable. We can transform it directly on the web via the JavaScript library, so you can send the Vega output to the front end and render it on the fly.

To get a sense for what it looks like, here’s the syntax for plotting a histogram of the number of radio stations within each market.

SELECT * FROM radio_markets
VISUALIZE Firms AS x
DRAW histogram
LABEL title => 'Radio Station Distribution Across Markets',
x => 'Radio Stations',
y => 'Markets'

This is what it looks like after rendering:

The part preceding VISUALIZE is just standard SQL. You can add WHERE, JOIN stuff, etc.

After VISUALIZE, we enter ggsql mode and start providing (approximately) the ggplot arguments in this SQL-ish syntax.

If you’ve used ggplot before, there’s this concept of “aesthetics” where we define the x-axis, the y-axis, colors, etc. VISUALIZE Firms AS x puts the SQL variable Firm into the x part of the aesthetic. You can also do VISUALIZE Firms As X, blah AS Y, blah2 AS color, etc, etc.

The DRAW clause tells it what kind of visualization to make (you can have multiple DRAW layers, of course), and LABEL… labels things (it can also change the tick labels, not just the axes, and alter any legends, etc).

By the way, to actually run this, I wrote a Python script. That’s clearly not the vision for how to use this. The idea is more of a notebook or other UI that can parse Vega output automatically to show the visualizations as output from an SQL cell or as results in an SQL-running UI, like Databricks/BigQuery/etc. But for messing around with it, you can do something like this:

import ggsql
import pandas as pd
import duckdb

df = pd.read_csv("radio_market_selected_30.csv", sep=":")

with duckdb.connect("radio.db") as con:
    con.sql("CREATE OR REPLACE TABLE radio_markets AS SELECT * FROM df")
    
reader = ggsql.DuckDBReader("duckdb://radio.db")

viz_query = """
SELECT * FROM radio_markets
VISUALIZE Firms AS x
DRAW histogram
SCALE y
LABEL title => 'Radio Station Distribution Across Markets',
x => 'Radio Stations',
y => 'Markets'
"""

viz = reader.execute(viz_query)

writer = ggsql.VegaLiteWriter()
vegalite_json = writer.render(viz)
print(vegalite_json)

And then, I just did:

python3 histogram.py | xclip -selection clipboard

And rendered it in Vega’s online demo thing: https://vega.github.io/editor/

[PS. Did you notice the source file is colon-separated lol? As best I can figure, given how long ago this was, I did it that way because the city names had commas in them (Peoria, IL), and I didn’t want to quote the strings :) ]

To show the SQL elements of this, here’s the same chart, deleting the long tail that starts at around 200 stations.

SELECT * FROM radio_markets
WHERE Firms <= 200
VISUALIZE Firms AS x
DRAW histogram
LABEL title => 'Radio Station Distribution Across Markets',
x => 'Radio Stations',
y => 'Markets'

The best way to explore a new tool and figure out if it’s really a sensible system worth adopting is to just try to make something more involved from the basic getting-started examples, without referencing the docs until you absolutely have to. What you’re looking for is whether things work logically together. You start to intuit the logic of a system (or lack thereof) when you force yourself to try and figure it out instead of just Googling/LLM’ing for “how do I…”

ggsql definitely passes that test. It’s natural and intuitive. Here’s a slightly more involved example to see what I mean:

SELECT * FROM radio_markets
WHERE Firms <= 200
VISUALIZE Area AS x, Firms AS y
DRAW point
   MAPPING Population AS size
   SETTING fill => null
LABEL title => 'Radio Stations and Physical Area',
x => 'Radio Stations',
y => 'Land Area (sq. mi.)',
size => 'Population (100k)'

My Takes

First off, this is super smooth, and I’ve always liked the “grammar of graphics” idea. It’s conceptually clear, easy to use, and easy to extend. It’s a great way to add plots to SQL. The part I’m stuck thinking about after playing with it is whether that’s what I want to do…

I can’t see this being very useful as a standalone thing. Even in a notebook, the utility seems limited. Being in SQL in-and-of-itself seems like it would only matter to the set of people who can be bothered to learn this new syntax but can’t be bothered to learn/don’t already know basic Python/R. I haven’t run a survey, but I’m pretty sure that’s an empty set. If I’m in a notebook, I already have access to ggplot/matplotlib/whatever, and I probably know how to use it.

One standalone use case that is valuable: it should make it easier to run visualizations on your main database compute engine, avoiding first loading the query results into your instance and then running Python/R/etc. You can return the Vega from the compute, plot it, and never touch the full dataset locally. That is super valuable if you’re plotting a larger dataset.

But the better use case really needs a full system where folks—or their Robots—write visualizations inline in a BI tool. ggsql (in its current state) isn’t a real replacement for the kind of viz you’d want in a BI tool, though. It’s not a language that can express (as far as I can tell) table viz, pivot tables, single value elements, layouts, etc. So, it’s got some of the elements, but not enough to replace a dashboard-like system.

On the other hand, it seems “right” to extend SQL with a declarative visualization layer on top of it. A good direction to go—and I have a feeling this is the plan—would be to add their grammar of tables as well to get a fuller system for expressing visual output in SQL.

Another thing I noticed was that I don’t think the VISUALIZE syntax is exactly a SQL extension in the sense that I don’t think it really has the same semantics or processing. Here’s an example I stumbled upon because this old dataset still has the base-R-like periods substituting for spaces in variable names:

import duckdb
import ggsql
import pandas as pd

df = pd.DataFrame({"numbers": [1,2], "variable": [3,4], "variable.dotted": [3, 4]})
with duckdb.connect("reproduce.db") as con:
    con.sql("CREATE OR REPLACE TABLE test AS SELECT * FROM df")

reader = ggsql.DuckDBReader("duckdb://reproduce.db")

## Quoting the dotted variable name works in DuckDB.

with duckdb.connect("reproduce.db") as con:
    print(con.sql("SELECT \"variable.dotted\" FROM test"))

### But not in ggsql
viz_query = """
SELECT * FROM test
VISUALIZE numbers AS x, "variable.dotted" AS y
DRAW line
LABEL title => 'variable ~ numbers'
"""
viz = reader.execute(viz_query)

writer = ggsql.VegaLiteWriter()
vegalite_json = writer.render(viz) # errors

I haven’t done a deep dive into the code base, but this very minor bug (?) suggests to me that ggsql sits outside the SQL engine and isn’t transparently working on the SQL elements. At least, that’s my first guess at why quoting wouldn’t work in its context. I could be wrong here, and maybe it’s just a bug on the Writer side (I filed an issue with the GitHub project, so maybe that’ll be all it is). It’s a small thing anyway.

Conclusion

In general, this is a nice solution to the problem of wanting ggplot in SQL. It solves that problem fantastically. Very clean. Very intuitive if you already know ggplot. I could guess how to do things after an hour or two playing with it, and I was usually right. So, that’s an incredibly strong win.

I’m less convinced it’ll become anyone’s daily driver unless it delivers a full BI experience. In the short run, I see the main use case being to create visualizations for large datasets that you don’t want to pull into memory. That’s a real use case, but a bit narrower than I was hoping for when I started playing with it.

As of today, I think it’d be faster for me, in most cases, to just copy-paste the query output into Google Sheets and plot it there if I want a quick visualization, but we’ll see where things go!

Things are happening—even in SQL.

Thanks for reading!

Zach

Connect at: https://www.linkedin.com/in/zlflynn/

Reading Club! April Edition!

I’m starting a new thing. I’m going to write about a recent paper that strikes my fancy once a month. More as a commitment device for myself to keep up with the lit than anything. I won’t just dive into the paper. I’ll talk about the relevant literature beforehand so the post is more or less self-contained.

This month’s paper is from a little-known, small journal from the fringes of Economics academia: “Econometrica.” It’s about figuring out how best to use the data to inform model estimates when we’ve got enough evidence to know the model’s “wrong.”

Subscribe now

It’s one of those excuses you’ll hear. “I know the model’s wrong, but all models are wrong,” and you, the listener, who is both smarter and better-looking, think, “Well, yes, but, like, clearly there’s a limit to that. If it’s really wrong about important things, that matters!” Wouldn’t you know it? There is a whole literature about estimation when we know the model is sort of wrong. Here’s part of it and the links!

The Paper: March 2026, Econometrica: https://www.econometricsociety.org/publications/econometrica/2026/03/01/Optimally-Transported-Generalized-Method-of-Moments

Free ArXiv Link: https://arxiv.org/pdf/2511.05712

Generalized Method of Moments

To grok the paper, we have to start with the kind of estimation problem we’re looking at. It’s a Generalized Method of Moments problem, a very common method in economics, but the thing about statistics is that many fields teach it and they all do it a little differently—mostly by historical accident. I think I’d like this framework, even if I wasn’t brainwashed into liking it by my education, but I guess that counterfactual is unobserved… It’s a nice, flexible way to look at the empirical implications of a model.

Suppose we have a vector of parameters b. The Generalized Method of Moments (GMM) thinks about problems where the parameters are identified by “moment restrictions” like this:

You can write a slew of common estimators like this.

• OLS: E[X(Y - X’b)] = 0

• IV: E[Z(Y - X’b)] = 0

• Maximum Likelihood: E[ (dlog f/db)(X, b)] = 0

• Quantile regression: E[X (p - 1(Y <= X’b))] = 0

• Etc.

The idea is that Models produce moments, so by finding the model parameters that satisfy the moment conditions, we identify the version of our model that matches the data.

For another example, in an older post, I used GMM to build this estimator that reduces the variance of quantile treatment effects in experiments.

Usually, we estimate the model by solving some version of this problem:

For some weighting matrix W. The estimator is consistent for any positive definite W, so the choice comes down to efficiency.

Overidentification

But sometimes models produce more moments than we need to estimate the parameters. This “overidentification” is useful because it allows us to use the extra moments to test whether the model is true. Intuition: a subset of the moment conditions pins down the model parameters, but we still have other moment conditions left over after fitting that we can plug the model parameters into to see whether all the restrictions can hold simultaneously.

But what if the model fails the test? What if it’s rejected by the data? It’s not terribly surprising that a model doesn’t hold exactly. So, how can we decide whether the model is wrong in the sense that it’s a simplification of a messy reality or if it’s wrong in the sense that we’re missing something critical?

One idea that emerged in the literature was: what if we could minimize the distance between our modeling assumptions and the data? We could capture some “pseudo-true” parameter, i.e., the most realistic parameter for the model given the data.

The parameter estimated by the GMM procedure above doesn’t really reflect that goal. It finds the parameter that minimizes the weighted norm of the moment conditions, which can have strong efficiency properties if the model is correctly specified. But if the model is wrong? Well, then the GMM estimator doesn’t really connect well with intuition about a parameter that minimizes the distance between the data and what the model requires. It’s just minimizing the norm of the moment conditions.

Empirical Likelihood

One idea in this vein is known as Empirical Likelihood, or EL. The empirical distribution assigns probability 1/n to each X(i), and we know this distribution doesn’t satisfy the moment conditions for any parameter b, but what if we take the mass points of the empirical distribution and tilt them until the moment conditions are satisfied for some b? What if we estimated the distribution of the data, subject to the constraint that it satisfies our model?

The empirical likelihood estimator does exactly like that. It maximizes the likelihood of the data subject to the moment condition:

Note that without the moment conditions, the solution to this problem is p(i) = 1/n, i.e., the empirical distribution. The non-negativity constraints can’t bind because the derivative of log approaches infinity as its argument approaches 0, so the first order condition is:

1/p(i) = lambda => all the p(i)’s are equal so p(i) = 1/n.

That means that if there were a b parameter that satisfied the moment conditions, we would just set p(i) = 1/n and not make any adjustments to the data. We’re effectively maximizing the log likelihood of the empirical distribution subject to the constraints of our model.

Even though there are n + dimension(b) number of parameters, there are nice tricks for computing the estimator from the first order conditions, i.e.

So,

Summing across i gives (from sum p(i) = 1 and sum p(i) g_k(X(i),b) = 0:

For any given “b”, we can then solve for mu via the equations:

Define the solution as \mu(b).

So we have this lower-dimensional estimation problem:

Optimally-Transported GMM

EL is what the maximum likelihood estimator of the empirical distribution would be if we added our model’s constraints, but this isn’t the only way to get at this problem. What if, instead of looking for the closest distribution to the empirical distribution by adjusting probabilities, we changed the mass points?

So, now we’re keeping the empirical distribution's even weighting but perturbing the data points to make the moment conditions hold, minimizing the distance we travel to “transport” X to Z.

Like EL, this problem also has a convenient solution despite being a high-dimensional optimization problem, but this time based on successive linear approximation, see Algorithm 2.1 in the paper.

The estimator is intuitive. It finds a distribution of the data that minimizes change, as measured by a clear metric that satisfies the moment conditions. You get numbers for Z(i), so you can start asking questions like: if X(i) had to change that much, do I think I’m really on the right path in modeling this? Or: yeah, that seems like it’s just a little off—fine. It’s a nice way to quantify how bad a problem the misspecification is.

Empirical Likelihood can do something similar, but to me, anyway, I have a lot more intuition about variable-space than about probability weights—your mileage may vary. “You’re telling me I have to make revenue numbers twice as large… weird. I don’t know if I believe this. No, I don’t think I do.”

Downsides here are that I’m not sure E[||X-Z||^{2}] is really the loss function we have in mind when we’re thinking about minimizing how much we change covariates—I don’t think it’s crazy or anything, but I think we’re usually thinking about something a little less… “cardinal”, is maybe the word I’m looking for. There are some variables that I can’t imagine being mismeasured, so I’m suspicious of moving them around. The method doesn’t really depend on the loss function here, but once you’re out of the least squares-ish world, you lose the nice computational trick, and you’ve got an n x dimension(X) parameter to optimize over…

One idea I had here was to make the problem a little more specific and, based on that, identify more intuitive restrictions for the particular problem.

For example, the overidentified linear IV model:

And the misspecification we’re worried about is that some of the IV’s are invalid. So, write:

And so, we’ve got:

Maybe the misspecification problem is that U = E + U* where E is correlated with W but U* isn’t, so if instead the world had produced U* as the structural residual we wouldn’t have any misspecification. We could solve for the minimum “E” that solves the problem, like so:

So now we have a structural interpretation of the error. If I had to give any modeling advice… knowing what the error is is the first step toward wisdom…

Anyway, I think there are many other interesting applications of this idea and Empirical Likelihood, but I don’t think I’ve ever seen it used in industry, despite seeing many examples of overidentified IV models rejected by the data. If you’ve got a GMM problem and the time, it’s worth taking a look at these methods to get a sense of the size of the misspecification problem, and, if it’s small, using pseudo-true estimates that balance the model and the data.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Once upon a time, there was a “Two-Language Problem” (which led to the creation of Julia; see the old texts). The problem was that while most analysis took place in high-level languages like Python or R, both were too slow, so most of the core logic of the various mathematical and statistical packages is not actually written in either language. It’s written in C (or in Fortran for many R packages).

Share

One of the main selling points of Julia was that it avoided needing two languages by using a compiler with multiple dispatch to mimic a lot of the niceties of a dynamically-typed, flexible language while still being performant because it could act more like a compiled, statically-type language (each function is compiled once types are known, so the code generated knows the data type).

I like Julia because multiple dispatch and the language's more functional nature fit mathematics and statistics well. I was never bothered by using two languages in a project. It’s actually kind of a nice experience to write the parts of an analysis that need to be performant in Fortran, and handle the data cleaning, etc., in R.

But I admit to liking C and Fortran because they were the first two languages I learned to program in. So, I’m biased.

And yet… like when Nixon said “We’re all Keynesians now”, we’re all two-language people now. The dream of one language has been deferred. The new dual-language world is Natural Language and X, where X is one of those old programming languages that requires you to think logically and carefully about what you’re doing (lame).

Before, the distinction was between a language where you needed to think in bytes and data types and a language where you could be a little more loosey-goosey about such things. Now, the distinction is between a language where you need to think in syntax and semantics and structure, and a language where you don’t.

We won’t be able to ditch structured language entirely. Natural language is a lossy representation of computing, just as the flexible scripting languages before it were. But it loses precision that can’t be recovered by spending performance.

The lossyness of Python and Ruby hurt performance. The lossyness of Natural Language hits business logic.

I predict a language will emerge that solves the modern two-language “problem”. It’ll be a more structured version of Natural Language that adds back some information, while still partially keeping the loosey-goosey feel of “build Spotify, make no mistakes.”

What I have in mind. Imagine you’re building a web app purely with AI today. You’re going to type lots of prompts to fix various things as you debug the app, but that doesn’t really scale for an app of any real complexity. Natural Language isn’t precise enough. In most cases, it’s quicker and better to just fix the code directly once the project is closer to being done.

But let’s say you defined things a little more precisely, then maybe you could keep it in Natural Language longer, i.e., in some kind of MVC-ish language:

• My app has a Model for users. They have passwords and emails and have to log in. Users can’t access other users’ stuff unless it’s been shared with them.

  • We need endpoints for users to sign-up, log-in, log-out,…

• My app has a Model for books. Users can write books or upload them as .epubs and share them with each other. There is an in-app reader for books.

  • We need endpoints for …

• Etc

So, I’m specifying the Model and a bit about the Controller components of MVC, and letting the LLM fill in what the views and controllers look like.

This scaffolding holds a lot more information than “clone Kindle.” I can add a model for pages read and mention that I want to bill by that metric rather than fixed prices, and I think the LLMs would do a decent job.

But this doesn’t really approach a maintainable language. I think we’ll get there. Someone will come up with a language that uses this newfangled LLM thing to both keep the expressiveness of Natural Language and recover some of the precision of structured language.

Or at least that’s where I hope it goes, because I doubt the old structured languages will ever gain market share.

(Alternatively, we’ll drown in endless AI slop, where nothing is built to last, and the past severs from the future, so that we exist in a never-ending present, neither returning nor progressing, until our souls, which yearn to plant some small monument, vanish into the void…)

I bet the language will look kind of like YAML, if I had to guess.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I love McDonald’s. I eat there 3-4 times a week—down from 5 pre-Covid, which is how we’re all marking time (from “since the birth of God” to “since the Plague”…). I couldn’t do that back in grad school (no money for such extravagances), but now that I have a real job...

Share

I’ve eaten just about everything on the menu locally, but they add things… If you’ve seen the awkward vid of the McDonald’s CEO eating a hamburger, you might’ve caught that what he’s promoting is the Big Arch, a new burger. The idea, if you listen to this great segment from the WSJ (free on YouTube) with the CEO, is to try to capture a slice of the premium burger market.

It’s going to work. Now, I know it’s in keeping with this blog’s usual thing to do an actual analysis with some math and a little theory, but this time I’ve got a better, more convincing reason. I tried it tonight, and I have news to report:

It tastes like the fruit must have tasted in the garden.

Dear God.

Homer sang of the Lotus Eaters, a people who were lulled into a state of perfect apathy where they forgot their homes, their goals, and their loves upon tasting the fruit of the Lotus and desired only to eat it and bask in the bliss it offered for the rest of their days. I now understand why such a life might be desirable.

In a world where everyone’s out there making the robots write their CRUD apps, it’s a breath of fresh air to see true innovation in the wild.

Let’s get into specifics:

• The patty is truly big. Much larger and thicker than you’d expect from McDonald’s. A bit off-brand in this respect, but it’s so well done. Delectable.

• They have these little fried onion things for some crunch. It crunches.

• The sauce is very Big Mac-ish, so if you’re a fan of that (like me), you’ll dig it.

• I was almost full before the fries (I eat in order: first burger, then fries. I’m not an animal).

This is my true and honest review. Sadly, McDonald’s did not pay me for this. If they want to send money, I won’t decline. I am 100% for sale. I want to be clear on that.

The Big Arch will be my go-to for as long as they have it. Fabulous work, gang.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

We all know that Data Scientist is a silly name. What kind of science doesn’t use data? Vagueness is the first sin of good science, and we’re putting it right in the name!

Share

I’m proposing a new job title: Table Reader. Now, I know what you’re thinking. “Table Reader is reductive. I don’t just read Tables! I do other things!” Okay. I’m sure you do. Really. I believe that is your truth.

By “Reader”, I do not mean a “Book Reader,” of course. I mean it in the Mystical sense. You aren’t “Analyzing” (vague and Latinate). You’re taking a Reading, tracing the palm lines of the Table and considering what the flight paths of birds might have to say (using weather as an instrument). Google Sheets has been missing a spiritual layer.

Data Scientist doesn’t say anything about what we actually do. Table Reader evokes.

A question arises. You consult the Tables for guidance, perform the Rites, Read the signs, and tell folks what the Tables say.

Each word is a strict improvement. Table is better than Data because it specifies the kind of data we look at. Data covers a lot more than tables, but the job doesn’t.

Reader is better than Scientist because it evokes Magic and Mysticism, which is both more inspiring and accurate. I’m not saying the Art has a lot in common with Divination, but it has more in common with that Practice than Chemistry.

For example, a computer is a Technology. If we both give it the same instructions, it’ll do the same thing. My nature and experience don’t affect the way it operates.

Magic is different. If the extensive literature on the working of Magic has established anything, it is that the Practice is particular to the individual. For example, Magic can be acquired via being born a wizard (Rowling, J.K. 1997), or you can be gifted powers by a strange lady who spends a lot of time in the woods (Map, Walter c. 1220), or be possessed by the Gods (Pythia, c. 1400BC), or learn to speak the language and meanings of the Nonmen, all but destroyed in the First Apocalypse, when the No-God, Mog-Pharau, rose millennia ago (Bakker, Scott R. 2004)…. we’re getting off track.

The point is that if we give the same Tables to two Table Readers, we’ll get different answers. It requires experience and intuition to know which signs are real and which are misleading, which models “make sense” and which don’t. So, Table Reading is an individual Art.

But, you say, “There is Math involved!” Sure, the Practice may use aspects of Science or Technology, like Mathematics, but consider:

If an Augur foretells the future by learning the biology necessary to identify the different parts of an animal’s entrails and from these gains insights into the length of Caesar’s reign, do we say he is doing Science? In part. But the Science is only necessary to construct the components of the Magic. The Science serves the Magic. So, clearly, Augury is a Magic.

In a similar way, the Science serves the Table Reader’s judgement as he or she divines whether the treatment is good or bad, whether to run this in logs or levels, whether we’re going to make next quarter’s targets or not, what kinds of fixed effects to use, and whether there are any plausible excuses to explain this product’s failure that should not obviously have been foreseen by you or your stakeholders (I kid, I kid).

So, we need a name that evokes Judgement and Practice, not Science: Table Reader.

I also considered “Table Teller”, but it sounded too authoritative, too crafted. In the good version of our profession, we merely read what the Tables tell us. We don’t instruct the tables, like the Medium doesn’t instruct the Spirits. We listen to Them.

Thanks for reading!

Zach

Connect on LinkedIn: https://linkedin.com/in/zlflynn.

We’re going to talk mainly about the general problem of understanding censored relationships, but specifics make generality easier to grok…

Share

Suppose we want to understand how the time until a Free/Basic Plan customer signs up for the Premium/Ultra/Mega Package varies with different actions or incentives the customer takes or receives. For customers that do eventually sign up for Premium, we observe the time it took them to do so from when they first signed up for the Free Plan. But for customers that haven’t signed up for Premium, the only thing we know about their Time to Premium is that it is greater than however long it’s been so far, i.e. we only have a lower bound on their Time to Premium. We’re missing data.

Most of causal inference is about making assumptions to fill in missing data. We don’t know what so-and-so would have done if they hadn’t been treated, so we find people who weren’t treated and say they’re kind of like so-and-so and fill in the treated subject’s missing outcome with theirs. In censored problems, like this one, we come up with an assumption about what the censored data would look like.

The classical methods (recall that “classical” is math-speak for “lame”) use parametric assumptions—impose a functional form on the censored variable’s distribution—to make the censored variable informative about the full distribution.

The classical example looks like this:

Where we only observe y* = max{y, 0} and y is unobserved. b and s are the parameters to estimate.

The parametric assumption, that y is normally distributed, allows the uncensored data to be informative about the distribution of the censored data because we know:

And we can identify the parameters of this distribution just from data on (y*, x). The normality of e allows us to connect the error from the uncensored data to what the error would be on the censored data. But this is a very strong assumption and we’re essentially letting functional form tell us about the extent of the censoring problem. Not very satisfying!

What is the most we can identify without this kind of functional form assumption?

What can we learn about how the distribution of a latent, censored variable changes with a covariate without making distributional assumptions about what we can’t see?

Fundamentally, the question of how the distribution of y changes with x is a question about how quantiles change. So, write:

f is the conditional quantile of y given x.

Let v be a uniform random variable drawn on (0,1). Then, here’s a random variable that has the same distribution as y given x = x:

So, if we can identify this quantile function, we’ll know a lot about how x affects y.

The problem is that we don’t observe y, so we can’t just estimate the model using our favorite quantile regression method.

Instead, we observe y* = max{y, 0}.

What we’re going to do is show that for a certain set of (x,t) we can identify f(t,x). Maybe that set is good enough that we don’t need to make any other assumptions. We can answer whatever question we’re asking the data just from identified points of the quantile function. If we need more, then we can extrapolate, i.e. make the dirty parametric assumptions, but we can do so, starting from points that are identified without having to make such compromises.

Whenever I’m thinking about how to identify something, I start with the distribution function. The reason to start there is that it fully characterizes the information available in the data. So it gives you the most identifying power to play with.

The data we have available to us is (y*, x). So let’s do a little algebra and see what y* given x tells us.

Let p(x) = pr[f(x,t) + e(t) ≥ 0 | x] be the probability of not being censored, conditional on x. The nice thing here is that we can estimate this function from the data because we know whether each observation is censored.

Let C be the event of an observation being censored. Then we can write the above as.

At all points on y*’s support, u is nonnegative because y* is nonnegative, so:

Let’s substitute in the true quantile function of y for u here:

Because f is the conditional quantile function, we know:

So substitute for that term in the conditional CDF of y* expression:

Now, we’re cooking. We’ve got the true quantile that we want (t) plus a bias term introduced by the censoring. It turns out that we can actually identify the set of (x,t) where the bias term is 0.

Suppose that for a certain (x,t): p(x) = 1 - t + c where c > 0.

Because:

We must have that f(t,x) > 0 because c > 0.

If the data is censored, then y < 0. Because f(t,x) > 0, the probability of y being less than f(t,x) given that y is censored is 100%. So:

So, for any (x,t) with sufficient probability of not being censored, we can identify the conditional quantile of y given x just from the conditional quantile of y* given x.

Cool. What does that buy us? This tells us the set of quantiles and covariate variables for which we do not face a real censoring problem and can identify the quantile function directly from data on (y*, x).

Unfortunately, this will usually not be enough. We’ll need to know how the quantile function responds at other (x,t) besides the ones that aren’t identified.

My preferred method for doing this is to do one of the following, depending on what you need: simply parameterize f(x,t; r) with a vector of parameters r. This form of parametric assumption is more natural and easier to motivate than assumptions about the functional forms of residual densities because you’re choosing how a response function varies as the quantile increases and as covariates change, and the shape is strongly informed by the actually-identified points instead of being a functional form assumption without strong justification, i.e. in a simplified example: suppose you had the following point-identified points and one non-identified point:

It seems sort of reasonable to think it’s roughly linear with x in this example so that x=4 should have a median around 4. The point is that you can use the nonparametrically-identified shapes to inform what parametric restrictions you put on the problem to identify the rest.

Anyway, this is just a tool that I think is less well-known than it deserves to be. So, I’m sharing it. Enjoy!

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Look. The first thing I’m going to do is hook you. I’m going to tell you this is about AI and LLMs. You have to believe me that we’ll get there. If, along the way, this starts to look like it’s about time series regressions (“boring shit”), just trust me. Believe me. Keep the faith.

Share

Computation and Statistics

Suppose we have a time-series regression (remember: trust me) where we don’t know the lag structure.

We find the “best” model for the time series in terms of out-of-sample prediction error by using the following computational procedure:

1. Fix an L, require that p cannot be greater than L, i.e. for every p greater than L, r_p = 0.

2. For each of the 2^L candidate models, compute the cross-validation score by leaving out future data and evaluating how well the model predicts it.

The reason we fix an L is that 2^L gets awfully large awfully fast. Computationally, we just can’t evaluate that many models. L models our computational constraint. We have two possible scenarios:

1. The optimal model selected via this criterion has max{p} = L.

2. The optimal model selected via this criterion has max{p} < L.

In scenario 1, we are computationally constrained. If we could increase L and make our model more flexible, our predictions would improve. We just need to call up NVIDIA and get more, better chips.

In scenario 2, we are statistically constrained. The problem isn’t that we don’t have enough computational power. The problem is that, given the data available to us, the best models we can construct are less flexible. We need more data before we can improve the model. NVIDIA can’t help us. If we spend billions of dollars on compute, the return on it will be zip, zilch, nada.

LLMs

The key question of how much core AI capability will improve over the next few years is whether we are computationally or statistically constrained.

Until now, LLMs have been primarily computationally constrained. When we got better chips, when folks spent more on compute, the result was better models. But I think anyone who’s used the tech over the past few years has noticed the core technology leveling off.

The difference between GPT 3.5 and 4.0 was enormous. I recall asking 3.5 to explain statistical significance in a stakeholder-friendly way, and it responded with gibberish that was so bad it didn’t even rise to the level of being wrong—and to a simple, textbook question! 4.0 was so much better that it boggled the mind. But 5.x… It’s fine. I’m sure it’s better, but I hardly notice.

I have no inside info, but it does make me wonder whether we aren’t moving into the statistically constrained regime—or, at least, will be in the next couple of cycles. And if we are… it’ll be very difficult to improve the core model. Throwing more parameters and more compute into the mix won’t change the mathematics.

Unlike other technologies, the returns to more data are not increasing. They decrease rapidly. You need exponential increases in data to generate sublinear improvements in performance.

I think we’ll run into the statistical constraint sooner than the market’s priced in. Word prediction is a hard game. The difference between the right word and the wrong word is a big thing.

The difference between the almost right word and the right word is really a large matter—'tis the difference between the lightning-bug and the lightning.

Mark Twain

In math, the loss function is steep. Small errors are big errors. A steep loss function, combined with a very high dimensional, nonsmooth problem, implies we’ll need a lot of data. We’ll have to get it from somewhere.

I think I can see how it happens.

Skills

We can’t generate vast sums of additional, real data. The AI companies are already using as much as is plausibly available. So, we need to make better data, not just more. Structured data. We have to add single observations that are worth billions of unstructured rows. We need to generate data for LLMs rather than rely on data originally generated for other purposes.

This is where folks are going with “skills” and related ideas. Provide tight context in a structured format to give LLMs data that is worth a vast amount of unintentional data.

Going back to our time series example. Suppose we knew that there was a seasonal component that hit every four time units. So all the models that can’t capture that structure are right out. This tells us information that might take a long time series history to discover from the data alone.

In other words, skills are parameter restrictions. If the restrictions are true, they dramatically improve model performance.

In five years, I think the vast improvement in perceived LLM performance will come from these kinds of purpose-built skills, not from core model performance. The core model will soon run into statistical constraints, if it isn’t there already, but there will be skills that help it write better code, better emails, etc, by applying rules and logic that humans already know in a structured way.

Like in classical statistics, the way to learn more from a fixed data set is to make more credible assumptions about the data-generating process. Give the problem structure by applying what you know about how the world works, and stop requiring the data to learn everything.

The trick is to tell the data what to think when you know what it ought to think and to let it tell you what to think when you don’t.

Anyway, I think that’s why it’s going where it’s going.

Fewer chips, more context.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Look at my website: https://zflynn.com