SLOBIL is a programming language. You’ve never heard of it because it has a shit marketing department (me). I wrote the initial version on bus 699 to DC in 2018–2019.

Why did you write it? What problem does it solve? Give me the elevator pitch.

I wrote it because I wanted to learn something. It solved my boredom. I never intended it to be useful to anyone.

There are so many programming languages, and they are fine. There is no good reason to make another one. Thankfully, there are plenty of attractive bad reasons. For example, making useless things is great fun and ennobles the soul… Curiosity and forty minutes of free time every day are good enough excuses.

I still use the language daily and add to it whenever it needs to do something else, or I just want to learn how to implement something. It has concurrency, even though I've never had a use case for it. It was just fun to figure out a nice way to do it (based on Ada, which I was playing around with at the time). The roadmap is whatever I want to read about and learn how to do.

This post tells the story of my now almost nine years of working on and hacking with SLOBIL, a programming language you’ve never heard of and should probably never use… I need to learn how to sell better.

Like all good stories, this one has a moral? Maybe.

Subscribe now

I wanted a language that had two features:

• The language should be easy to use interactively, i.e., the REPL should be convenient enough that a “program” in the language could just be a set of functions you load into the environment.

• I had this vague idea of making a “dictionary” the core data type. This eventually transformed into the prototype-based object-oriented system that SLOBIL is centered around, but that wasn’t there at the beginning.

Given these two requirements, I didn't start planning and fleshing out the language—that’s work, and I was distinctly not about that for this project. I just started writing code to see what I would come up with.

Code is a much better language for designing a program than English, anyway.

These two requirements implied the language should have very limited “syntax.” It should have very few special forms, and it should look “command-ish” to be easy to use interactively. I imagined it would end up looking visually Tcl-ish, but with LISP-y semantics. So, I came up with this as the basic structure.

OPERATION DATA DATA DATA ... DATA .

Where the OPERATION is some built-in operation of the language, and DATA are various values. The idea is that this is basically like a normal shell command, except it ends in a period because I hate newline-ending code.

The first step was to write a few basic data types like INTEGER (initially just ints, eventually bigints from GMP) and REAL (just C double), and add some basic arithmetic operations.

Now, I could write things like:

add 1 2 3 .
-> ans = 6

I played around with my over-engineered calculator and realized the next problem: how do I type in 5 x (2 + 3)? Okay, so of course I could do:

add 2 3 .
-> ans = 5
mul 5 ans .
-> ans = 25

But that seemed cumbersome. So, I added the first piece of syntax. A way to group substatements. I decided to use square brackets like this:

mul 5 [ add 2 3 . ] .
-> ans = 25

The square brackets work by executing all the statements inside them (you can put as many statements as you want), and then the argument equals whatever is currently in the ans variable after those operations finish. The arithmetic operations set the ans variable to their result, so this worked naturally.

Next, I started thinking about how to assign values to variables. The first thing I realized was that if I wanted to keep the regular syntax, then variable locations needed to be first-class data types. So I created “Slots”, a data type that refers to the location of a variable.

Slots aren't limited to particular types. Any type of data can be put anywhere. I needed this to be true given the “ans” solution above, because “ans” is a particular Slot, and it needs to hold any data type because all operations write their return value to it. So the language became strongly typed and dynamically typed. I didn’t initially have an opinion about which direction to go on typing, but the decisions along the way converged on that as the solution.

A major plus to using Slots is that the data locations are data themselves, which enables neat tricks. It lets us transform and manipulate data locations instead of the data itself, which can improve performance and simplify code.

Another advantage of Slots is that we don’t need a special form for assignment. I added a set operation to take care of assignment:

set /x 2 .
print x .
-> 2

The key idea of “slots” is that you refer to the location, the slot itself, by preceding it with a forward slash (the actual symbol went through several iterations, including $ for a while, but I’m skipping to what the symbol ended up as). You access the data stored in that slot by just omitting the forward slash.

For example,

set /x /y .
print x .
-> /y
set x 3 .
print y .
-> 3

Notice how the second set doesn’t have a forward slash in front of the x. That means we access the actual value stored at Slot x, which turns out to be Slot y's value. So the set operation assigns Slot y to 3.

Now that I had a concept of data location, I reworked the native operations from hard-coded lookups into first-class data types, inserting them into the required slots.

That's when the language’s structure started to come together. I realized I was really creating a dictionary of data stored at Slots. Clearly, it would be useful if this “table” was its own data type. Eventually, I would call these objects… Objects. Because I’d written Objects initially as just the global scope that contained the mapping between Slots and Values, Objects function as both scopes and data in the language. So, you can pass scopes around to higher-order functions that run code inside certain contexts, etc.

set /env [ object /x “Hello” /y “Goodbye” . ] .
print x .
-> Error: No data found at /x.
in env ( print x . ) .
-> Hello

At this point the language had developed a personality, a recognizable way of doing things that just sort of emerged as I kept solving problems incrementally.

I don't think it’s a bad way to start a new project. Start with the project’s core premise and key constraints, then keep solving problems to get to the next step. Solutions that solve problems while respecting constraints survive. The bad ideas die off. Creation by evolution, not design.

It reminds me of the way you prove things in mathematics. You start with a few premises and try to figure out how to get somewhere useful. In the end, we write it up as Theorems and Proofs, as if we knew what we were trying to prove from the start, but that isn’t how you work on it.

At this point, the language still didn’t have that key element of structured programming: functions. So I had to add another data type.

I thought it would be interesting not to attach an environment to functions. Instead, the data type (“Instruction”) would just represent literal code that could be executed with arbitrary Objects giving the context. They know nothing about the environment in which they are defined.

set /add-one ( add t 1 . ) .

The key thing you can do with instructions is call them. Because they are just code, they don’t have an environment, so you have to provide one, like this:

call add-one /t 4 .
-> ans = 5

This calls the instrument inside an environment where the slot /t is set to 4.

Of course, it would be annoying to have to type “call” a lot, since calling Instructions is something any well-structured program will do a lot. So, I added a little sugar to the syntax. I expanded the normal OPERATION DATA DATA form I’d been working with. Now, the first item in a statement did not need to be an Operation. It can be some other data type. When it is another data type, a particular default operation is inserted implicitly as the first element of the statement. So we can also just do:

add-one /t 4 .
-> ans = 5

This led me to think that this kind of syntactical trick would be useful for other data types as well. For example, set would be a common operation, so why not just assume it if a statement starts with a slot, i.e.

/x 2 .
print x .
-> 2

I realized I needed a third kind of parenthesis. I had one for substatements (immediately executed code) and instructions (code as data), but I realized I needed another type of substatement to write conditional statements naturally: lazy substatements. So, I got the new rule that anything between {} would only be executed whenever the value was actually required by the operation. This enabled us to write if statements because now:

if True { print “Hello” . } { print “Goodbye” . } .

Prints Hello, not Goodbye, because the third element of the statement is accessed only if the second is True, and the fourth only if the second is False.

By far the most-used SLOBIL program of all time is the time-keeping program I wrote to track hours for consulting work. This program led to several syntax conveniences being introduced to the language as I got more day-to-day experience with the language’s quirks.

It looks like this:

You can type:

client .

And get a menu of current clients to select from, then you type:

start .

At the end of the day, you just type:

report 2026 5 3 .

To give you a report of how many hours you spent working for each client on 2026–05–03.

I’m writing about SLOBIL in this here blog, even though it has nothing to do with Statistics and Economics, because it’s one of the few projects I’ve worked on for almost a decade. Most of what I do is ephemeral. An analysis for so and so to decide if we want to do such and such, which dies as soon as the decision is made, like a male spider post-coitus, its purpose fulfilled… okay, I need to work on less disgusting similes. Maybe a little model that lasts a few years until it no longer applies to the biz.

But I’ve worked on this useless, silly little programming language longer than I’ve been married. Every now and then, when I think of a little thing to fix or improve, I boot up Emacs. It’s a nice way to pass the day. I wrote a bit of it before the rehearsal dinner.

I started writing it well before the LLMs blew up. I still haven’t set the robots on it. They’d undoubtedly improve it.

But it’s not meant to be improved. As I’ve been saying, SLOBIL is useless, fit only for being a fun thing to tinker with, when the mood strikes.

The Robots make programming feel like work, like you’re trying to “accomplish” something. You can’t tinker with Robots. They press you to Accomplish Tasks (tm).

I knew nothing about how to write a programming language or the concepts you’d have to figure out (what is the internal representation of code objects?), but I learned by realizing either the first way I tried was horribly inefficient, or it just didn’t work.

I wrote the parser by hand. Practically, it made sense to leverage flex and bison, but it was fun to figure out an algorithm that worked well.

I kind of wonder if I would’ve bothered to learn all that if I could've started with a skeleton by typing, “write me an interpreter for a language that looks like this…”

I’d like to think I would’ve. It was a good time. But who knows? It’s hard to resist the temptation when you’re tracking down memory leaks.

If you’re curious, the full documentation for SLOBIL is here: https://zflynn.com/slobil/slobil.html, and you can download it from GitHub.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Enclosed are some thoughts about causal inference that I had on September 4th, 2026, at approximately 1:30 a.m., Chicago time, the best time for clear thinking.

Subscribe now

Every causal inference problem is about how you’re missing some data, and every solution is about ignoring that fact.

“Causal inference” is sexier than “missing data.” Missing data evokes “data cleaning,” the least sexy task in any corporation, a task best handled by someone else.

Causal inference evokes telling Other People that the chart they like so much isn’t causal. You can tell little stories like “this chart is like saying ice cream sales cause summer”—which is fun, and lets you embrace and nourish the natural smugness required to enter any scientific field.

[If the Youth just changed the construction “it’s giving [adjective]” to “it evokes [adjective]”, they, too, would sound like an urbane, sophisticated critic…]

But causal inference is no more than a special type of missing data problem. We know how the world works under one policy or customer decision, but we want to know how it’d look if they’d faced a different policy or made a different choice. We’re missing data on what could have been.

There are both infinite ways to attempt to establish causality and very few methods, like there are more novels than anyone could read in twenty lifetimes, but so few stories.

Say we have a treatment and we know which treatment each unit experienced (Z). We want to know how it changes the distribution of an outcome (Y). For each value of z, we can define the “potential value” Y(z), which indicates what Y would be if the unit had received a different treatment. We observe what actually happened, i.e., Y(Z) = sum 1(Z = z) Y(z). We only know what happened for one treatment.

1. Write down what you know.

2. Write down what you want to know.

3. Bridge the gap.

So, what do we know? We know the distribution of [Y(Z), Z]. We observe what actually happened for the treatment assigned to our units, and we know which treatments they were assigned.

What do we want to know? We want to know the distribution of Y(z) for any particular treatment z. We want to know what the world would look like if folks in our population made different decisions or faced a different environment.

Now, let’s bridge the gap:

We can write the distribution of Y(z) as:

P[Y(z)] = P[Y(z) | Z = z] Pr(Z=z) + P[Y(z) | Z != z] (1 - Pr(Z=z))

This is just a basic fact of probability distributions: the law of total probability. It doesn’t tell us anything. It cuts things into pieces, so we can see them clearly.

The key is to start building the bridge on the ground you are sure about. The thing we know is the distribution of Y for the treatments people actually received, so we’ve got:

Pr[Y(z) | Z = z],

because that is just the distribution of Y for units that actually faced the treatment z. So, we can just measure that from the data.

We also know which treatments the units received. So we know Pr(Z = z).

What we don’t know is just:

P[Y(z) | Z != z],

and that’s all of causal inference.

We don’t know this factor because we’re missing data on Y(z) for the population that doesn’t receive treatment z. All of causal inference is about coming up with different assumptions that tell us the value of this term, despite not having any data on it.

Of course, we have an alternative. We could refuse to make assumptions. Always consider the outside option. There is a way to restrict P[Y(z) | Z != z] that doesn’t depend on us taking any leaps of faith at all. We know:

0 <= P[Y(z) | Z != z] <= 1,

which tells us that:

P[Y(z)|Z=z] Pr(Z=z) <= P[Y(z)] <= P[Y(z)|Z=z] Pr(Z=z) + (1 - Pr(Z=z))

Which isn’t a lot, but it’s something. You have to start somewhere. In fact, if only a little data is missing, the gap between the bounds is fairly narrow. If most people in the population of interest receive treatment z, then we’ve got strong identification from the data alone. But if, say, 50% of the population doesn’t see it, then the gap between the lower and upper bounds is 50%—which is huge, when we’re talking about probability distributions.

The Assignment Assumptions impose restrictions on how treatments are assigned, allowing us to use the data we have for Y(z) from the Z=z population to translate it into a number for P[Y(z) | Z != z].

If you’ve ever run an experiment, what you’re relying on is the following Assignment Assumption.

Random Assignment. Suppose that treatment assignment is random in the sense that it is independent of Y(z).

If Random Assignment is true, then conditioning on Z=z or Z != z doesn’t change the distribution of Y(z) because it’s independent of assignment. So, we can write:

P[Y(z) | Z != z ] = Pr[Y(z) | Z = z]

And we can measure the right-hand side! It’s the same as the first factor and, in fact:

P[Y(z)] = P[Y(z) | Z = z]

When random assignment is true, correlation is causation.

A natural extension of Random Assignment is to suppose it holds conditional on some other variables.

Conditional Independence Assumption. This assumption says that, conditional on some X (which we observe), Y(z) and Z are independent, i.e., P[Y(z) | Z=z, X=x] = P[Y(z) | X=x].

This is what you’re relying on when you run a regression “controlling for X” or when you use inverse propensity score weighting or when you use a matching estimator or… It’s used a lot.

The assumption is, roughly: Well, yes, there’s selection, but it’s determined by something I know about. All we have to do is condition on a few things, and viola: we’re back to random assignment.

It isn’t a very credible assumption. The idea that there’s really nothing we don’t observe that plays a major role… I’ve never really heard a convincing argument for the strong version of this assumption. The best genre of justification is along the lines of: we know there’s one huge selection problem, and I can control for it. Are there other problems? Sure, but I’ve knocked out one big one. It’s not a bad argument as things go, despite the complexity of interpreting the conditional independence claim correctly as you add more and more X’s. And we shouldn’t discount the good—no, great—thing about this assumption is that you usually have the data to use it. You can always regress Y on X and Z, and it really does eliminate some forms of selection and mitigate others.

There are other assumptions in the Assignment category, I suppose, but the ones that give point identification are all—more or less—different varieties of these. We argue there’s some way to treat our data as an experiment, and then we go from there.

I wrote a few ’grafs about other approaches, but then saw myself veering off the tracks, and the post stopped being a blog post and veered dangerously close to something worse. A textbook. So I snipped it.

Besides, it’s all just variations on the theme.

Here’s a good exercise. If you’re learning causal inference and want to be able to write recipes instead of just reading the cookbook, answer the following question about any standard, workhorse methods you learn:

• What does it say P[Y(z) | Z != z] is?

• Does the method even estimate P[Y(z)] (or E[Y(z)])? One trick the “causal inference” methods use is to change the question to get their answer. Causation is hard. Sometimes you have to cheat. They switch to “treatment on the treated” or localize the treatment effect to a specific subset of the population, etc. It’s fine, really. Just watch out for the trick where these changes suddenly get reversed in the policy analysis, and we’re back to treating the estimate as a global treatment effect.

• What does the data alone tell you about whatever parameter the method’s actually estimating?

• How does the method fill in the gaps?

And so on. Answering these questions tells you what’s really going on.

The downside of grokking the methods is that it’ll make you critical. It’s very easy to spot problems in a causal analysis because it’s a hard problem. You’ll find flaws in every analysis.

The cure to this disease is being forced to do it yourself.

Like any positive theory of how to learn something about the world, causal inference has a nihilistic response. There are two species of nihilism in causal inference:

1. All the assumptions are wrong. We can learn nothing!

2. All the assumptions are wrong. So, it doesn’t matter what we assume! We don’t need to justify it.

Nihilism is very uncool.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

If you want to play around with my pricing survey app, check out MK*SRVY. There’s a video walkthrough and post about it:

For the past few months, when I’ve had free time on nights/weekends, I’ve been writing this delightful little survey makin’ and analyzin’ tool that I dubbed MK*SRVY (pronounced: Make Survey). I was inspired by the pain of creating pricing surveys in other tools. Now, you’ve got a handy-dandy way to write pricing surveys for all the products. The video does a little walkthrough of how easy it is and how flexible the built-in analysis tool is. It’s neat! And it’s free!

…Free-ish, naturally. I’m trying to become a mogul here.

I thought I’d charge for Workspaces, to easily collaborate with others to ensnare massive corporations—who else can afford $10/month (with a 7-day free trial)? You also get Conjoint analysis in the premium plan, and MK*SRVY doesn’t return silly things like “partworths” or whatever nonsense the other survey platforms thought up (seriously: it’s silly and you end up having to do the conjoint analysis you paid like thousands of dollars for yourself). MK*SRVY returns choice probabilities for certain feature combinations when they compete against other feature combinations, i.e., it tells you what demand is when facing certain competitors (or the outside option).

The name for the Premium subscription was obvious: MK*ULTRA. This made me chuckle, is kind of related to understanding human behavior—like a survey—and is therefore excellent, expert branding. Some branding folks might caution against using the name of illegal government programs, but, mercifully, their advice is too expensive, so they didn’t get a say.

It can, of course, do more than pricing analyses. The same question type used for Gabor-Granger-style willingness-to-pay analysis could be used to optimize free trial lengths, discounts, etc.

In any case, if you feel so inclined, check it out, give it a spin, and let me know if any issues!

Because this is a tech-ish blog, I’ll give the “stack” here for the curious:

• Ruby on Rails on Postgres

• An R package I wrote to do all the analysis

• Conjoint analyses use STAN

• Optimization Problems use a Global Optimization routine over compact sets.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Check out MK*SRVY!

I bring good news: Slop isn’t going to win. Sure, the Reddit Regressions spew it on demand, cheaply and at scale, but sheer volume isn’t enough to be competitive.

It’ll lose—at least in software—for an Economics 101 reason: saving on the fixed costs of production does not give you more pricing power, and when higher levels of fixed cost purchase quality, a high quality entrant can survive an endless sea of low-cost entrants.

Slop can only come to dominate those industries where it reduces variable costs. It still might not. But there’s a chance…

Subscribe now

To simplify the problem: suppose there are two potential firms that can choose their product quality, their prices, and whether to enter the industry at all. We want to understand when the whole industry can devolve into Slop.

Let’s start with the bad case.

If firms can create higher quality products by increasing their variable costs, then (Slop, Slop) is plausible because low-cost Slop allows you to charge lower prices that a high-quality product can’t match, potentially excluding the high-quality entrant.

Firm profits are, for a quality choice u, where c(u) gives constant variable costs, q(*) is demand, and f is the fixed cost of entry:

If the slope of c is steep enough, then higher quality entrants will have to maintain higher prices to have the positive margins needed to overcome fixed costs. This leaves room for a low-cost product to enter and acquire price-sensitive customers, further driving down margins—possibly to such an extent that high-quality entry becomes unable to overcome the fixed costs of entry (f) and we get (Slop, Slop).

This model doesn’t describe the Software industry though. The variable costs of software don’t increase with product quality (well, except the newfangled token-powered LLMs). The vast majority of quality-relevant costs are fixed costs: the engineers writing the code.

So, now suppose c does not depend on quality but instead product quality is financed by spending fixed costs (costs that do not increase with scale) so that we have:

Where S is market size, scaling up the size of demand.

Now, margins cannot be squeezed by a low-quality entrant because the pricing decision does not depend on fixed costs. So, (Slop, Slop) becomes untenable in a sufficiently large market.

To see this… Suppose there exists a non-Slop u* such that monopoly profits are positive. Now, say a Slop entrant comes in. At equal prices, everyone will buy the non-Slop product and the non-Slop product has the same marginal cost as the Slop entrant, so there will exist a way in the second-stage pricing game for the non-Slop product to make positive variable profits. Then, in a sufficiently large market, the non-Slop entrant will be able to earn enough variable profits to offset the cost of entry, preventing (Slop, Slop).

Because pricing is untethered from the cost of quality, high-quality firms have strictly more pricing flexibility and better demand curves, so they cannot be forced out by Slop entrants.

Okay, but what if there are a lot of Slop entrants. We might think that the sheer number of Slops must be able to take enough share away from the high-quality firm that it can no longer remain viable.

This turns out to not be true. The high quality entrant can maintain its share above a floor level in the face of endless cheap, low-quality competitors. I won’t spoil the model. You’ll have to read it from the source because it’s a delightful read:

Shaked and Sutton (1983)

This analysis actually shows one industry where Slop might win. LLMs themselves.

Because for LLMs there is a significant variable cost to product quality, so low-quality LLMs can offer prices that the high-quality Reddit Regressions can’t match.

The Creators of Slop being consumed by their Creation… It’s poetic and will, therefore, happen. The universe conspires to make good stories come true.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I have written this blog for three years this month. First, on Medium before it had a name; then, on Medium when it was called Good Enough Statistics and briefly flirted with the idea of accepting submissions; and now, a shiny Substack, where I plan to stick around. Because it has LaTeX embeds… [and I can write where I live vicariously through the blogs of the kinds of people that know the fashionable places to eat in New York City.]

Subscribe now

The nature of the idea distribution is that the good ideas are much better than the so-so ideas, which are astonishingly better than the bad ideas, and the frequency of such ideas is inversely proportional to the quality. My rare—but extant!—good ideas since writing this blog are spread across the years, and so, they’re hard-to-find in the chronological Archive.

I decided to make a guide, so you don’t have to sift through the shit.

The criteria were simple. If I thought the post was pretty good when I scrolled by it in mid-August 2026, then I added it. I call this criterion “Taste (August 2026)” because I’m hearing from all the well-connected folks that Taste is the future of humanity.

So, here are the good posts. They are in a rough, chronological order because I couldn’t think of a better way to organize them. Enjoy!

I look at how we can use experiments to optimize a product over time when the product design decision can be modeled as a continuous variable. For example, prices, discount percentages, the length of a free trial, or the period of time a one-time purchase remains in effect. The key idea is to treat experimentation as evaluating a function and its derivative at a given parameter value, and updating as if you were solving an optimization problem via gradient descent.

I’ve made this point in a few places, but I think this is the most succinct version: we should choose metrics to maximize surprise. We don’t want to goal our experiment on metrics where we know what will happen. We want to maximize what we learn. We want to think, prior to the experiment, that it could go either way. For example, conversion rate should never be the primary metric for an experiment that gives some users a discount. Of course, larger discounts will increase the conversion rate. We don’t need to run the experiment to know that. We need a metric that measures both the potential costs and benefits of the change we’ve introduced. This shows up more subtly in many experiment designs discussed in the post!

A simple production model with uncertainty about how best to do a thing and a technology to automate doing it makes surprisingly strong predictions that match at least my experience with LLMs. The idea of the model is to treat LLMs as an automation technology (of course, in reality they have other uses) and study the implications. A neat result here is that workers will use LLMs for tasks they can do faster/better without the tech for moderate-uncertainty tasks (perhaps, LLM-generated docs…).

There’s this common objection to running an experiment: we need to move fast here. I argue that, properly understood, experiments do not slow us down. They increase product velocity. Velocity, of course, is the product of speed and direction. The steering wheel affects velocity at least as much as the gas pedal. It will take us longer to change direction if the change doesn’t work out if we don’t run the experiment because we won’t have evidence there’s a problem for a while.

I reference this post all the time. It’s one of the things it’d be cool to have more general buy-in for. I use insights from a paper that derives frequentist hypothesis testing as a statistical decision rule to form the basis for a simple, practical way to choose significance levels, instead of defaulting to 0.05.

The context for this post was the prevalence of blogs around this time saying statistical significance was bad, and I just had the thought: is it really? I didn’t find most of the alternatives in these posts obviously better, and I started thinking, “There are all these posts about how bad statistical significance is, and yet everyone uses it. Curious.” So, I started thinking about the revealed-preference gap, and I concluded that statistical significance is a pretty good way to make decisions. It’s better than the alternatives usually proposed anyway. It has the two main elements: effect size and how uncertain we are about it and a single-parameter to tune how much weight to place on both of them. Clearly, any decision rule will be some version of those two features, and statistical significance has an intuitive weighting scheme (the significance level). So, I came away with the sense that the criticism of statistical significance is about something else, not the statistical significance framework, and discussed creating a Cult/Church, depending on your perspective.

We don’t test ideas we think will do poorly. We spent engineering and design time building out the feature because we think it's a great idea. So, without experiments, we would just launch our great, new idea. If our new idea is actually great, then all the experiment did was waste time. If we hadn’t run an experiment, we would have launched earlier. But if, to our surprise, our new idea is actually bad (sadly, this is usually the case—at least in its initial form), then the experiment is highly valuable. It prevented us from making a mistake and blindly pursuing the idea we were once so gung-ho about. I include a calculation of the value of an experimentation program based on the losses it prevents.

This is a really common situation: so many experiments run on a site simultaneously in a mature experimentation program. So, the question is: do interactions between the experiments matter? I do a little math and show that—quantitatively—the interaction effects need to be very large before the effects could actually lead to you making the wrong decision. And that’s the main point of experimentation: decision-making. Interaction effects do impact estimated effects, though, if you’re relying on the exact effect value in some way. There’s no avoiding interaction effects—or environmental effects present during your experiment but not afterward—from biasing the impact estimates. Online experimentation is not a clean-room laboratory, but it is an amazing way to make the correct decisions about which direction to go.

Sometimes, you’ll get the advice not to “peek” at your experiment results early. But peeking is good! It can reduce the expected time of experimentation! Sometimes good things happen, and we should roll with it. You just have to make inferences in a way that accounts for peeking, and it’s easy.

This is one of those problems that rarely shows up in academia because the dataset is more or less fixed, so the issue is under-discussed in standard statistical training. But it’s very common when running online experiments. We have a metric for a unit we’ve randomized, and the question is: “Over what time period should we compute the metric after the unit enters the experiment?” The default in many experimentation platforms is from the time the unit entered the experiment to today. This causes a lot of problems for inference. It’ll make your experiments take much longer to run. It is even possible that you would never find a significant result—even if one exists—if you continued to run the experiment forever!

Thank you for reading, and for a delightful diversion from work for the past three years,

Zach

Connect at: https://linkedin.com/in/zlflynn

Subscribe now

Why has LeBron James made so much more money than me? He doesn’t know much about stochastic processes, causal inference, or industrial organization—at least, as far as I can tell.

Because LeBron James is a scarce resource. There isn’t anyone quite like him. You can’t replicate him by hiring the next best alternative. There’s a huge drop-off. There are a lot of people like me. A dime a dozen in college towns. You can buy yourself a Zach on any street corner fairly cheap. He’s a commodity, really, just without enough demand to trade him on the CME.

So, you want to be the scarce resource—rare, hard-to-find, collectible.

I doubt the trick to make yourself employable in the Robot Future is to learn how to use the Robots well.

For one thing, they’re easy to use, and the Robot companies have every incentive to make them ever easier to use.

It’s easy to learn AI. The trick is to think to use it. “Oh wait, I can just type that into the computer now.”

Most of the increased productivity I’ve got from the LLMs looks like this:

“Please search the folder for bugs. Report any bugs you find. Just report them. Don’t edit the code.

<Bugs>

<I fix the bugs>

“Continue searching for bugs.”

Anyone can do this, and any problem that can be solved cheaply by a technology that everyone knows how to use will not demand much of a wage.

You want to do the opposite, of course. You want to know something that most people don’t. Call it “the return on secrets.”

So, you learn hard things, the kind of things that take years to learn, because no one wants to spend years learning. The difficulty is what makes the skill scarce.

It’s not a theoretical law—not even close—but an empirical regularity: technology tends to complement the productivity of the human input.

[Armchair theory: technology complements human productivity because humans innovate, and we create what we imagine we will use. There’s probably a very smart economic history book on this that I’ll have to read (I’m currently working my way through this one, which I highly recommend).]

To convince yourself this is still the case today in the world of LLMs, take the problem space they’ve improved the most: computer programming. If you’re not a programmer, try to write an app by burning Claude credits. Then, have a programmer do the same thing, but they can only use Claude prompts. The programmer’s result will be so much better that you will wonder whether you even used the same tool. It’s not because they know how to use AI better. They know how to program, so they’ll create better programs, regardless of medium.

It turns out that it still pays to know something—which sucks, of course.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Subscribe now

The first rule of a good model is that it’s explicit. It goes step-by-step.

Isn’t a model. It doesn’t say anything. It doesn’t tell me how the world works. Good models that change people’s minds explain things. They tell us how the whole world works, step by step, tracing the path from point A to point B. The players have incentives, goals, and structures they have to operate within.

Simple equations, although usually more robust (e.g., regressions, treatment effects) in the sense that they are true in more or more plausible states of the world, lose to models because they purchase their credibility at the cost of failing to provide meaning—“it made revenue go up” not “customers who value feature X have weaker price elasticities and…”—and we want to understand.

For example, we might compare two price vectors p and p’ in an ideal experiment. We can pick which one is better by comparing, say, average revenue, but you’ll be surprised how unconvincing that is unless there is some model for why p’ ought to do better than p. If you can say p’ takes into account the substitution patterns between the products you’re selling, while p prices each product as if it were independent, you’ll get immediate buy-in and even a thought that we should apply this everywhere, to other product areas, etc. If it’s just… empirically better, we’ll ship the treatment, but we’ll be wondering why constantly. If things go south, doubt will creep in, and we’ll start to suspect it was a fluke.

This is why religion crushes simple non-belief in the marketplace. It explains. It tells us how the world works, a real model of things. Non-belief doesn’t do that. You have to add to it to get a model of the world. Credibility doesn’t stand a chance against a model. No one’s overthrowing the bourgeoisie because they don’t believe in God. They have to believe in something else first. The dialectic. It’s important to consider these things when designing models to pick the optimal promo depth for your e-mail campaign.

The point is that building a model of what you’re doing matters. Customers choose among these alternatives. They value such-and-such. And, look, if that’s true, then the right price is $53.92 because… etc, etc.

So many good insights fall out of models. You can figure some things out deductively. The data is limited, and if you make it solve everything, it loses the power to tell you the answers to the truly unknown questions. For an extreme example that’s hopefully illustrative:

Suppose you have two products:

1. You can read an unlimited number of books from a certain library, and the reading app’s interface is Gold.

2. You can read two books per month from the same library, and the reading app’s interface is Blue.

Look, the product’s got two features, and some people might like Blue and other people might like Gold. So we’ve got both horizontal and vertical differentiation. Who’s to say which product’s price should be higher?

But we know, right? We have a model of the world that says readers care more about the number of books they can read than the color of the UI. We have another model of the world that says, in fact, we should just let people choose the color. No one’s willing to pay anything for that.

We could test this, and statistical noise would give us various customer valuations for Blue and Gold. It really would. We’d start talking about how people value Gold at around $2.31/month. There’d be confidence intervals. It’d look like analysis. We could make plans to drive Gold upsells.

Or: we can admit we know things, and if we use statistics to try to learn what we already know, we’ll just be drawing random shocks and tap-dancing around the truth.

The thing we don’t know here is how much people value reading two books per month versus the unlimited option. The signal we get from the data here is worth the noise. We can’t just know this from having a basic model of the world.

If instead of two books, it was a million, then we’d be back in the world of modeling again. No one could physically read that many books in a month. We don’t need to test. We have a model of the world—linear time—that collapses these two products into one.

When you get more explicit with even very simple modeling, surprising things fall out. Like, we have a suspicion that it takes us longer to use the Robots than it used to take us to do some tasks, but we use them anyway. Why? Well, it turns out a very simple production theory makes exactly that prediction…

We’ve got limited time, limited data, and infinite questions. We have to choose what to ask. Models help us decide what’s worth asking and what’s not.

Thank you for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Subscribe now

A take with no equations? Sounds risky, but let’s give it a shot…

In the ancient past, biz writers read their documents more than all other readers combined. LLMs changed that. Now, the Reddit Regressions frequently generate memos, design docs, PRDs, etc, etc with only light editing by the responsible parties.

I don’t like it.

1. All Change Is Bad, and the kids really must get off my lawn.

2. Biz writing is not about the document itself. A newspaper article or a book is its own end, but that’s not really the case for the memo. The point of writing it out line-by-line is to think through the problem and its solution step-by-step. Maybe other people will read it and be convinced, but primarily, the author organizes their thoughts, folks add a few comments, and then it serves as the basis—an outline—for a live discussion where the decision actually gets made.

The traditional biz doc represents, if not an engrossing read, how a person thought through a problem. It might lack all the other elements of “good writing,” but it has the most important one: the author wrote it to say something, and they thought it through.

The LLMs make every doc, wiki, PRD, etc, read like a high school essay on The Scarlett Letter—naturally, because they are written for the same reason: a box must be checked. The main difference is that the pre-LLM high school essay usually has an endearing personality and a visible human struggle to make word count: “I think it is bad what they did to Hester Prynne, really. I certainly wouldn’t like it were it done to me, the Golden Rule and all…” The LLMs have that repetitive, exhausting quality sans voice.

Writing it out is the best way to convince yourself that the idea you had really works, when you consider all the information carefully. When drafting, you end up thinking things like, “Oh, wait, I forgot about the subscribers who are still grandfathered into the 2007 plan.” Writing moves information from RAM to Disk, freeing up working memory to analyze your 2-am-brilliance critically.

“Write an argument for…” or “Write a PRD for a new feature that does X, Y, Z” are silly prompts because if you haven’t written it out, you haven’t convinced yourself the argument or plan is true or a good idea.

“Tell me why this rough doc is wrong…” is a much better prompt. The Robots have good thoughts once they are dealing with real output, and are great at catching questions you forgot to ask.

They’re terrible at generating biz docs from a paragraph and data—which isn’t really a knock on ‘em. They got a paragraph prompt and had to write a whole essay, and their topic isn’t an important 19th-century novel, but the packaging of an HR SaaS product.

It’s a tough assignment.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

There’s an inverted U-shaped relationship between competition and innovation.

With no competition (including no serious threat of entry), there’s little incentive to innovate because you just extract monopoly rents, and there’s little extra profit to make by making a better product, past a certain quality level.

Under perfect (intense) competition, any product improvement gets competed away, driving profits back down to zero, so you get no returns on the investment you made to improve the product.

This is the justification for patents in, say, drugs. Any new drug can be cloned cheaply, creating intense competition post-innovation, so there’s no reward for investing all that dough in developing the drug. So, we reduce competition with patents.

You need just the right amount of competition to get strong innovation. Not too much. Not too little.

Subscribe now

I wonder if this isn’t an argument that the LLMs will end up getting government money.

It sounds silly—and I tend to be reflexively against this kind of industrial policy—but there’s essentially zero property rights. No lasting product differentiation. The lagging models catch up to the frontier models in a matter of months, which is not material for current use cases. They produce (significant) economic value, of course, but I think there’s a real doubt the LLM producers can capture it, given the market structure.

For now, folks are still betting that there will be a winner of the race. But if this market structure sticks around too long, without any shakeout, investors won’t continue to believe that. They’ll start to believe there are no profits out there to offset the scale of the investment.

There is some return to packaging, software, harnesses you might bundle with the LLM. I might like the ergonomics of Claude or Codex more, but that’s small-fry differentiation compared to actual model performance. I can’t imagine the returns to this kind of product differentiation are enough to encourage the investment needed to make Silicon Gods.

When products are valuable, but the ex-post profits will be competed away, we tend to do… something. Patents or other restrictions on entry are the most common approaches, but they won’t work here—at least not so far as the US Government is concerned. China will keep producing these models, economic returns be damned. The Chinese models can’t be easily excluded from valuable markets, and the models are valuable…

And so, if someone can’t break away and competitive profits are all that await Anthropic and OpenAI, I’m betting we’ll end up subsidizing investors to encourage future investment. Yes, it’ll make this whole corner of the industry a little more… mucky with the usual rent-seeking, and the incentive problem of encouraging riskier investments, but I could see the government reasonably deciding it’s less bad than the alternative.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Subscribe now

Award-winning directors and actresses alike need business advice from esteemed corporate drones, like me. I wish it weren’t the case, but it is. Both of these ladies have failed to fully and completely cash in on remake/sequel/prequel/branding fever, and they have such a golden opportunity.

Lost in Translation 2.

One might think a sequel is unnecessary and, in fact, destroys the very arc of the movie, the ambiguity, the aftertaste. It is not that you are wrong. You are worse than wrong. You are not asking the right question. The right question is: will this shit sell? The answer is: Yes. With the right script. With the right direction. With the right actress reprising her role.

The people cry out for it.

Look, I’ve even sketched the plot for you. Picture this, Sofia.

I walk into your office, a bundle of nerves, my harried look a dead giveaway that I’m here to pitch you. You’ve heard it before.

“I have a pitch for a movie,” I say. You roll your eyes.

“It's Scarlett Johansen playing the older version of her character from Lost in Translation. She didn't know what to do after Japan, so she went to law school. She went to Yale after all, majored in philosophy, thought she could be a writer. Where else does this type end up? It's only natural. She has to do something, and there's law school. Something.”

You lean forward in your chair, intrigued. Finally, someone else saw what you saw…

“She's working a case, a morally questionable one. I don’t know. She’s defending clubbing baby seals in the Arctic. But it doesn't start there. It starts with her defending something more innocent, but still guilty. And then it's just a little more each day. She meets a guy in private equity working on an acquisition. It’s New York, okay? She’s divorced that photography guy because he didn’t support her going to law school. ‘I just want to make something of myself,’ she said. ‘So does everyone. But a lawyer?’ He didn’t understand. They drifted apart. Everyone's mad at the new (much hotter) private equity guy because he and his firm laid off three entire Midwestern towns stocked with the most sympathetic people. Scarlett and hot finance guy bond over being hated and fall in love, and then the rest of it is a rom-com where they tell jokes about clubbing baby seals and firing people while dodging protestors and having red paint thrown at them. It's really not their fault. Why shouldn't they be in love? It turns out because they are conflicted on an upcoming acquisition. They are on opposite sides. She’s representing another bidder for this distressed Midwestern amusement park company, and they lose touch because they can't communicate until the trial. And then she remembers that she wants to be a writer, so she writes a romance novel about it, working title Conflicted, and she tries to get it published, but no one wants it because it doesn’t ‘speak to the market’.”

“Dear God,” you say, evidently impressed. “Please leave.”

You need a moment to collect your thoughts to respond to my artistic brilliance. I understand, but I offer one more nugget to convince you.

“The movie ends with a flashback to the first installment of the Lost In Translation universe. The part where Bill Murray whispers in her ear. You can hear the audio clearly, and Bill Murray says, ‘Look, I’ve done art. I’ve been an actor. Now look at me. I’m reading whisky ads in Japan. Just take the LSAT, kid. Just take the LSAT.’ And they kiss.”

“Security,” you say.

Thank you for reading, Ms. Coppola and Ms. Johansson, and for your consideration,

Zach

(I am currently on Day 1 of 2 of unemployment. I should not be left with nothing to do.)

Subscribe now

Dashboards are the worst form of self-service analysis, except for all the other forms that have been tried from time to time...

They are terrible because they are inflexible. Dashboards take material time to build, yet they don’t answer 85% of the class of questions they were supposed to answer. The questions are ever-changing, while the dashboards are static, fixed. They need another filter. Another cut. Another unit. Another metric. So, what becomes of the dashboard? Reactively, we expand its filters and parameters. Still, it sits there unused, and folks continue to ping their friendly neighborhood analysis person to do whatever needs doing.

And then: what does the analysis person do? They don’t send back a dashboard-like document, a sentence of text around charts and tables. No. They write a memo [a “doc”], describing what they found and what the analysis means.

The point of dashboards is to enable self-serve analysis, yet they look nothing like the analyses data scientists produce. Think about how weird it would be if a data scientist just sent you back a couple of graphs when you asked them to look into something. There should be a narrative, an argument, facts… Something.

Why should self-serve analysis look different than the analysis I do?

The only difference between self-serve analysis and bespoke, artisanal, handcrafted analysis from the Analyses region of Northern Illinois should be who ran the thing.

Self-serve analysis has a great premise. The business questions come from one set of humans, but the answers traditionally require another set of humans, who (on occasion) have to eat, sleep, and play Cyberpunk 2077. What if we could scale the answers? What if we could write the analysis once and scale it so that we could automagically generate answers to a broad class of questions? If only Eve hadn’t….

So, this is my hope for the BI Tool of the Future, and, because I’m an Optimist, my hopes are the same as my predictions: we need a Memo-Generating Machine.

A Memo-Generating Machine does the following:

1. Within a certain class of questions, it translates the details of a questioner’s ask into code that returns an answer.

2. The answer is a memo, explaining how we answered the question, why we did it that way, and what the analysis shows.

I’d been toying with how to make the mystic machine real for a while, and then, suddenly, it became commonplace. In 2019-2022, I wrote a bit about this version of BI, and made a Rails app that implemented a rough version of the idea. It worked okay, but you had to do a decent amount of the memo generation yourself as the analyst, which made it difficult to automate for more open-ended analyses. So, it didn’t really solve the problem very well. But things have changed since those dark days…

You’ll notice, of course, that the Memo-Generating Machine is more or less Claude Cowork with well-specified instructions.

Claude doesn’t generate dashboards when you ask it to find the answer to some question you have. It writes memos, the standard output of data analysis, like a good little robot.

So, I dig what LLMs have brought to the data analysis scene because, despite their flaws, they have at least identified the correct form of the output, which is the most important thing.

The one downside of LLMs is that they’ve enabled a lot of truly shitty data analysis, and, in typical tech fashion, they have done so at scale.

The problem is they do not put structure on the form of the analysis. Often, they get some context on the database structure and how to run queries, and then they just kind of react to prompts, generating queries—often incorrectly—in subtle, difficult-to-detect ways, or they simply answer the wrong question. A lot of the folks who want to run self-service analyses can’t really tell when the LLM does something wrong or silly because they need more context about the database or statistics, etc, so it’s not really a self-service structure on its own.

You need a lot of context to do a strong analysis, and it’s unlikely to be provided by a generic solution, like a metadata layer on top of a database, because it’s not just the data that matters but how to analyze it.

So, a raw conversation with an LLM merely backloads the usual ask to the analysis team. Instead of “can you pull this?” the question becomes “can you check this?” Which is an improvement, but it’s not really self-service.

The good news is that it doesn’t have to be this way. One structure I’ve been experimenting with is being heavily prescriptive about the form of the analysis but lightly prescriptive about the form of the inputs.

Something like the following structure for, say, a Claude skill:

1. Tell the little robot to extract some info from the user’s prompt…

2. Tell it to run this exact script to do the analysis, filling in the parameter gaps as needed from the prompt. Here, I insert an exact R/Python/SQL/Shell script that does whatever analysis I’d do for this class of question directly into the skill.

3. Tell it to write a report on the analysis, focusing on …, noting …, recognizing that X is true, and giving it instructions for interpreting things, etc.

4. Feed it some of my artisanal, handcrafted memos and tell it to write like me, a human, not like it, a robot.

Step 4 is still a very much unsolved part of this version of self-serve analysis. Steps 1-3, the status quo robot does reasonably well. It does a great job of extracting parameters from natural-language input and asking follow-up questions if the user didn’t include some relevant info. It always follows instructions to run the exact script. It mostly does a good job of including the relevant constraints on the memo, but oh boy, does LLM writing suck.

It’s horrible. Worse than lifeless. I wish it were what we used to call “robotic.” Unnatural, overly stiff writing would be so much preferable to its imitation of a high school book report. Insufferable adjectives. Why does it insert imagery, metaphor, and simile in contexts where they make no sense? And so much of it. So much of it. They write, and they write. I think it’s because Anthropic charges by the word.

Things haven’t changed since Dickens*. Everyone responds to incentives. Great writers and robots alike.

* Dickens was not actually paid by the word. His tokens were “installments,” which had to be roughly 32 pages. Still, naturally, he wrote to generate installments.

AI-generated text exhausts the reader.

I have faith we’ll fix this, and, in the meantime, this version of the Memo-Generating Machine works. Highly recommended.

Aside from the writing in the memo itself, the other problem to solve with the current Memo-Generating Machine is cost. Running an analysis now has a variable cost beyond the querying and compute costs, and it can add up. Dashboards do have a significant advantage here because they’ll usually materialize a table once and run various, relatively cheap transformations on this dataset. The Memo-Generating Machine does much more analysis on the fly, so it’s more expensive (but more flexible), even before accounting for token costs.

That said, the more tightly constrained analysis setup above greatly reduces costs compared to having the LLM create its own queries or whatever. They tend to use a lot of compute, process tons of data, and burn tokens while the robot backs into something approaching a reasonable analysis via an expensive iteration process.

That’s basically what I’ve found works for cost control. If a skill is too expensive, constrain it and allow less discretion.

The “script-constrained” LLM doesn’t use nearly as many tokens, though, as the standard vague prompt to an LLM with access to your data warehouse, so it’s dramatically cheaper than that increasingly popular option.

Anyway, this is where I think BI tools are headed. Away from dashboards and towards memos. Which was always the right direction for things to go. So… good news! There are still problems to solve to make it work swell, but I’d guess this structure is the standard form of self-service analysis in the next five years.

/remind-me 2031-07-10

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Tesla caps per-employee AI spend at $200.

Tesla—and other companies—have started limiting AI use at the employee level. The theory behind capping individual users—which I buy—must be that each additional dollar an employee spends on LLMs generates less value for the biz, so limits should help prevent employees from wasting the budget on low-value actions that cost more than the value they create.

They’ve “clearly” read my blog post showing that, left to their own devices, workers use LLMs even for tasks they could do better without them. For example, reformatting documents or generating fancy slide decks are two mostly unnecessary tasks that AI can handle in the background, but it’s probably not worth the high token cost.

Subscribe now

A sort of obvious LLM budget allocation strategy, if you believe labor and LLMs are complementary (which seems natural to me), is to allocate budget based on the employee’s salary. I’m sure this would go over… horribly, but it makes sense economically.

Like, let’s say total output from a certain worker with productivity parameter A who can spend up to M dollars on LLMs is:

Which, when theta is less than 1, captures the idea of decreasing returns to additional LLM spending as workers spend it on less and less valuable tasks.

Then, we can maximize net output from that worker by solving:

Which gives us:

So, we should assign a larger budget to more productive workers, and in a competitive labor market, productivity is well proxied by wages. So, we should have a regressive AI budget, with larger allocations to higher-paid workers and lower allocations to lower-paid workers.

I don’t expect many companies to do this directly even though it makes a lot of sense from an efficiency perspective. They’ll do it indirectly, of course, via allocating different amounts to different job families or levels.

It’d be cool if a company experimented with LLM budgets. Randomly give some workers more of an LLM budget and others less. Then, you’d get a decent read on the average value of tokens. It’ll be difficult to get a lot of power unless you’re a large company… like Tesla. It’d be cool to get the return on AI spend (ROAIS?). If a company does this, please publish the results. It’d be super interesting how the return on tokens (ROT… okay, this one’s unlikely to catch on) compares to their price in corporate land. The context is pretty important here: mature companies aren’t doing a ton of completely new, green-field projects. Most work is fixing or improving stuff. So, the ROT might not generalize to vibe-coding, etc, but it’d be super useful for companies figuring out what their budgets should look like.

If you’re reading this and need a way to justify doing an experiment like this to your employer, just remember that you can share the results of this experiment with OpenAI or Anthropic during the next contract negotiation and, if the ROT is low, probably convince them to give you massive discounts.

For my own personal AI spend trend… I have used LLMs less and less throughout 2026—which is sort of the opposite of how I’ve adopted most new technologies. Usually, as I move further up the learning curve, the tool becomes increasingly valuable to me, so I use it more. But it actually takes a remarkably long time to get, say, a query right with LLMs. I’ve accepted that it’s almost always easier and quicker for me to just write it, even after factoring in having to DM someone about exactly what column ABC means.

I still haven’t really found a way to use LLMs effectively for business-process-type tasks either. They generate unusable, overly dense documents and presentations with far too many images or too much formatting. Give it to folks plain, simple, and true. Write directly. Just say it. Don’t present it. You are hereby banned from using adjectives ever again. Do not editorialize. No matter what prompt you give it, it just doesn’t understand how to Be Normal (tm).

On the more positive side for the tech, I’ll never debug without AI again. It’s so incredibly strong at getting the details right when checking my code that I run basically everything I write now through the prompt “Check this for bugs. Don’t fix the bugs. Just report them.” Life-changing.

I’m not sure how well my use cases generalize, but something I’ve noticed with “generative” AI is how bad it is at generating stuff compared to evaluating stuff. If I give it a claim or some code, it’s very good at figuring out the problems from the context I’ve generated. But if I want it to make a claim or code itself, it’s a mess without significant elbow grease that I’d rather spend just writing the first draft myself and using the robots as the fanciest of spell-checkers.

I currently allocate $0 to LLMs in my personal life, aside from whatever I get from Gemini for having the “premium” Google account, which I mostly have for the extra storage, because Codex’s free tier is far too generous. I debugged this entire side project (about 10-20k LOC Rails app) easily using the Codex free tier.

I feel bad about extracting all that value from OpenAI without paying them a dime. Not bad enough to pay them, of course, but I’ll give them an in-kind payment: free pricing and packaging consulting. You should either make the free tier less generous or just kill it and offer a $5-10 plan with low limits. You’re welcome.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

The Internet is aflame with an accusation that DOGE (and, thereby, Elon Musk) is responsible for the deaths of millions because it cut funding for various aid programs. The claim comes from a study, but this post isn’t going to debate the study because it’s not relevant. I’ll just take, for the sake of argument, as given that aid cuts cause deaths and that quantitatively everything in the study checks out.

It wouldn’t surprise me if the causal story is true, but the nature of the discourse given that is bad, and it touches on interpreting policy research, so it has, however tenuous, a connection to what this blog usually studies.

Subscribe now

Every large government policy like this has mortality impacts. Every single one. If you cut taxes or raise them. If you require stricter medical ethics rules. If you criminalize drugs or make them legal. If you decide to spend X on aid, instead of 2X.

How many people did the folks who decided to fund USAID at its prior funding level kill because they didn’t fund it at twice that level?

It’s surprising that most of the defense of DOGE has so far focused on problems with the study, etc. That’s not the point. Policymakers cannot and should not choose policies that minimize worldwide mortality, and, certainly, if they do not implement such policies, they are not “responsible” for the deaths caused by not implementing the maximally mortality-reducing policy.

The Argument (a publication I generally like, for what it’s worth) says it directly in this article, right in the lede:

Elon Musk really doesn’t want you to say he’s responsible for the deaths of millions.

So yes, that the folks behind DOGE are responsible for these deaths is clearly what people want to argue in the Discourse. The phrase is probably overused, but there is a “motte-and-bailey” vibe to this where folks make the strong claim that Elon Musk is responsible for the deaths of millions and then they retreat to the claim that X people died because of aid cuts. These are different claims. Moral responsibility for death is clearly distinct from passing or not passing various laws or funding various programs.

I’m sure I could’ve saved incremental lives by donating a larger percentage of my income to some life-saving program. I’m not responsible for any deaths. That’s silly.

People die because the speed limit is 70 mph instead of 45 mph. Is every local politician who keeps the speed limit at 70 responsible for the road deaths? What about everyone opposed to the prohibition of alcohol?

It’s definitely a useful, worthwhile thing to do to study the mortality impacts of a policy. It’s part of analyzing policy impacts—alongside other impacts!—and as a study, there are no issues with this. But we shouldn’t take such analysis and use it politically to parcel out responsibility for various deaths to policymakers. It’s a never-ending war where the different sides produce excess mortality studies for each of their opponent’s policies.

Hot take: I think it’s a bad idea to make our politics one where we regularly accuse our opponents of murder if they cut funding for a program.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Today, we’re breaking from this blog’s usual topic (statistics, economics, etc) to indulge in a far more important topic: a hobby project I built seven years ago. Aside from being a trip down memory lane, it’s possible this blog has a point. Two of them, in fact:

1. Doing things just because is a perfectly good reason to do things, and

2. Extensible double-entry accounting apps are a valid form of artistic expression.

Subscribe now

It was the Fall of 2018, and I was to be wed in the Spring. I did not have much in the way of savings in those ancient days, and weddings cost money. I needed to budget for it.

Naturally, I needed to write an extensible double-entry accounting system to do this. It had to be programmable in Scheme, of course. I cannot overemphasize how much all of these requirements were unavoidable design constraints. It is impossible to budget for a wedding without Lisp.

And so, leas was born. Leas is an extensible accounting system that you can use interactively in the terminal. It’s extensible with Scheme, a full programming language, embedded in the app.

[I wrote it without thinking about LLMs—it was 2018—but now that we are in the new world, these tools do make it even more useful. “Codex, write me a script in Guile Scheme to update my home value based on whatever Zillow says it is and record the incremental value as home equity gains” just works like magic. Try getting a web app to do that.]

The initial idea was to make a tool that fit in between two kinds of accounting tools.

1. ledger. A tool that produces reports from specially-formatted text files. It’s very UNIX-y. I like ledger. It is highly extensible because you just need a script that modifies a text file. Easy. I actually first toyed with the idea of making leas an interactive tool to modify ledger files, but I didn’t like the result, and I didn’t find ledger very convenient to use in its natural state. I wanted something more interactive that was still extensible.

2. GUI-tools like GNUCash. I used GNUCash in grad school, but it was a separate GUI app that I’d always forget to open. I wanted something quick and easy from the terminal. GNUCash had some extensibility (in Scheme, too) but not super easy to use.

I also just thought the idea of an Emacs-like personal finance tool was neat-o.

So, I came up with this little structure:

• Interactive C-based command line app.

• The prompt was just a Scheme interpreter that called various Scheme/C functions exposed to Scheme.

My first critical design decision: I realized right away that I did not want to type parentheses all the time:

(command arg1 arg2)

So, I made the interpreter insert the outer parenthesis automatically so that this Just Works (tm):

command arg1 arg2

But, I wanted the program to be more… interactive. I didn’t want to have to type full commands. So, I settled on a UX like this:

> spend
5: Groceries
6: Rent
...
To Account: 5
1: Checking
...
From Account: 1
Amount: 109.23
Description: Walgreens
Year: 2026
Month: 6
Day: 22
>

Of course, I could just code this interaction in C, but I wanted to make the interactive system sort of “universal” and something you could use in Scheme extensions to add new interactive commands. So I came up with this system for making any Scheme function interactive, and also added some special data types like this:

(define add-record
  (lambda ()
    (leas/call "leas/at"
               (list
                (cons "Account" "current_account")
                (cons "Amount" "real")
                (cons "Description" "string")
                (cons "Day" "day")))))

So you can use leas/call to provide arguments to any non-interactive function using data types like “real” and “string” but also with some leas specific datatypes like “day” (for a date), “current_account” for the currently selected account, “account” for an account, “expense_account” (for an Expense account), “liability_account” (for a Liability account), and so on. So now we have some actual accounting data types.

This little structure let me write a small set of core routines and data types in C and then most of the interactive structure in Scheme (the UI, more or less) which was nice.

The basic structure was:

• Book

  • Account

    • Transaction

Each Account has a Type. One of:

1. Asset

2. Liability

3. Income

4. Expense

Buying a McDouble, taking out loans, paying them back, and earning money are all just transfers between accounts. You transfer funds from the Income account to the Asset account on payday. You transfer from a Liability to an Expense account when you pay with a credit card or from a Liability to an Asset account when you take out a loan.

There is this concept of the “current” account, a pointer that attaches to a certain account and marks it “active”. That makes it possible to use commands which just take the current account as their argument which makes it quicker to navigate around. For example, lt lists the last few transactions in the current account.

I use leas every day, so the thing has ended up having lots of niceties. There are commands for doing things like paying off a loan (two transfers: one from an asset account to an expense account to pay interest and one to the liability account to pay off the principal).

I’ve written Scheme scripts to automatically update the values of any funds/stocks/etc. I hold by scraping the internet.

I should point out that this whole “budgeting” thing actually worked. Go figure. Keeping track of spending did let me kind of thread the needle on the wedding. It reduced stress, increased confidence, and gave my skin a healthy glow.

The whole thing is only about 2000 lines of code. Simple, but I’ve used it daily for a long time now. I don’t commit much to the repo anymore. The software is finished, more or less. I fixed an edge case in it a couple of weeks ago. I’m averaging a commit every two years since 2020. That’s the nice part about non-SaaS software. You can finish it. Nothing changes in the environment, and it’s not like any bug is a big deal. It’s running on your machine. You can’t steal someone else’s info if I make an error in the controller.

Of course, there was really no point to this project. The number of budgeting and personal finance tools out there is endless. They are better documented, tested, and so on. I could’ve found something that worked well enough to not be worth the hours I spent on it.

But I had oodles of fun, and there’s the pride of making something useful.

I don’t know shit about fixing cars, so I have to find something else to do.

There’s this terrible piece of advice folks get. If someone’s already solved the problem, don’t re-solve it. Think about another problem worth solving. It’s bad advice. You can’t solve any problem until you learn enough about it.

The best way to learn how the wheel works is to reinvent it.

Oh, and here’s the Github for this project: leas.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

This post will be about experimentation, but there’s a wind-up.

Today, we’re going to look at a class of experimentation problems that we might call “tuning.” A problem where there is a wide variety of plausible choices for a parameter, and we’re trying to find the best one among nearly infinite plausible choices.

Common parameters to optimize include:

• Price

• Promotional Discounts

• Hyperparameters of ML models (search, recommendations, etc.)

Subscribe now

Suppose you want to maximize a function f(u,t). At any point in time, you know f(u_t, t), but you can’t evaluate f at any other points but whatever u happens to be “active”. For example, you know your current prices (u_t) and you know revenue today, but you don’t know what revenue would be at other prices.

How do we learn the best u if we can only evaluate one u at a time?

Let’s think about it as an optimization problem:

Suppose f is strictly concave in u and its gradient satisfies a Lipschitz condition (we’ve decided to be formal-ish today—suit and tie). Then, we could solve this problem via gradient descent, i.e.:

For some suitable sequence of (a).

The problem is: we don’t know the derivative of f, and we don’t know how to evaluate f at other points.

The trick is that we can use experiments to iterate on the gradient descent problem. An experiment is just like computing the objective function and its gradient. It takes a few weeks to compute, though, and our data limits how many points we can compute at once.

The starting point is the relationship between treatment effects and derivatives in this context. Suppose u is a number (not a vector), and you expose half the population to u and the other half to u+h. Well, viola, you have an estimate of the derivative:

[You can expand this to multi-dimensional u by adding more arms to the experiment or by adding structure to f that lets you learn other derivatives in a neighborhood of the current values of u you’re testing.]

First, suppose f doesn’t depend on time, and also, for the sake of simplicity, u is not a vector. Then, the optimal u doesn’t change. It’s static. A stable target you’re trying to hit. Using the gradient descent analogy, we can try to find the best u like so:

When the treatment effect from increasing u is positive, increase u. When it is negative, decrease u, and so on.

There is a vast literature on step sizes that work for gradient descent with a strictly concave f. The key property we need is a method that doesn’t require us to evaluate the function at many points because experimentation limits the number of points we can evaluate. A straightforward approach to choose a, assuming f is strictly concave, using a modern “two-point” method from the literature that works for general concave functions, c/o Malitzky and Mischenko:

Where Df_t is the current derivative estimate, and Df_t-1 is the prior derivative estimate. I’ll call this the MM-rule.

[The MM-rule formula treats dividing by 0 as infinity, so you’ll always choose the other branch of the min{*,*}.]

The main advantage of making this choice instead of the classical gradient descent step size is that you don’t need to run experiments with a ton of arms to approximate the maximization problem:

You’d operationalize this like so:

1. Initially: Run an experiment with the status quo “champion” u vs a “challenger” u + h. Choose h large enough that you’ll see some real action statistically, but small enough that you’re reasonably approximating a derivative. Choose an arbitrary a to get a new challenger. The right choice here is a bit of an art, but it’s fine to just choose it so the treatment looks “reasonable.” This is just an initial condition.

2. Do this again to have enough initial conditions to start using the MM-rule.

3. Now, run the best arm of the previous experiment versus the new challenger generated by the MM-rule.

4. Keep iterating, using the MM-rule above. Repeat step 3 forever.

The nice part about this method of experimentation is that it allows you to make careful, small but continuous iterations toward the truth instead of making large, costly attempts to jump to the optimal value (usually based on strong functional-form assumptions). You’re always testing your latest proposal against the data, so you can’t drift too far.

Practical issues you’ll run into: the formula for a might produce too small of a step size for you to see any real action. The treatments are too close together. It’s fine to put a lower bound on step size, even if it violates some guarantee of convergence. The precision the optimizers look for isn’t attainable if we’re estimating functions from noisy data.

Now, what if the function’s value changes over time, i.e., we have f(u,t)? That’s why we’d run experiments in the first place, instead of choosing different u’s and trying them out to trace out the function. We need to run experiments because otherwise we’ll confuse variation in different u’s for variation in t.

When we experiment, we learn:

So, we can trace out the derivative with respect to u from the data, as in the static function case, but the optimal u we’re chasing is changing as well.

I’m proposing using the same algorithm above to solve the problem when t changes the function (and so, the optimal u)—with a small twist.

Of course, if we’re completely general about how t changes f, this approach won’t work, because the optimal u could jump around wildly, but that’s not really what we think will happen.

Essentially, we’re going to assume that the optimal u varies “slow enough” with t that we can track it by accounting for changes in the gradient with respect to u.

The main practical change is that the MM-rule now depends on function values that we can’t just keep in memory. We need to evaluate the gradients of f(*,t), not the gradients we previously computed of f(*,t-1). So, here’s my simple, practical iteration strategy:

1. Run a four-arm experiment. The experiment has u_t (the current iteration), u_t-1 (the previous iteration), u_t + h, and u_t-1 + h (the h’s can be different if you’d like).

2. Use the results of this experiment to construct L_t.

3. Now, if you look at the formula, you’ll see that technically it depends on a_t all the way back and prior L_t’s that are function evaluations will have been evaluated on f(*,t-2), for example, instead of f(*,t). My sort of pragmatic approach here is to ignore that and use the old a_t’s where they appear as they were when you computed them.

I think this is a reasonable, practical tradeoff to avoid exploding the number of arms the experiment needs while still accounting for the fact that the function has changed, moving from t-1 to t.

Experimentation can test more than button colors. We can use it to understand the models that drive the modern biz. Models that target certain customers with promotional offers or specifically tailored marketing strategies. The number of reasonable models is vast, and finding the right one requires forcing proposed models to confront the truth about their performance. Experiments provide the ground truth and structure the search. It’s a nice pairing.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

All analyses rely on assumptions to map datasets to conclusions. The data doesn’t speak for itself. You have to shape it. Ideally, the mold you fit the data into is based on a reasonable theory of how the world works.

Subscribe now

Analysis is a trick to, with enough data, reduce the world to two potential outcomes:

1. The analysis’s conclusion is true.

2. OR: the assumptions of the analysis are wrong.

This is very useful. It moves the conversation forward. Folks can’t just disagree with your conclusions. They have to disagree with your assumptions, too.

Therefore, an analysis is persuasive only if your assumptions are more plausible than your conclusions.

If your assumptions are more plausible than your conclusions, then disagreeing with the conclusion becomes more difficult after the analysis. Before I could argue against the conclusion directly. Now, I have to argue against the assumptions, which are harder to dispute.

But if your assumptions are less plausible than your conclusions, then you haven’t convinced me. After the analysis, I know either the conclusion is true, or your assumptions are wrong. If I disagreed with the conclusion before, and it is even easier for me to disagree with your assumptions, it is just as easy to disagree with you as it was before your analysis*.

There’s a famous paper in empirical labor economics that illustrates this.

Card and Krueger (1994) claim to show no effect of a minimum wage increase on unemployment.

They do so using a difference-in-differences analysis of bordering counties in Pennsylvania and New Jersey after New Jersey raised its minimum wage. The idea is that these counties are similar to each other and face similar shocks, except that the New Jersey counties received a higher minimum wage. Card and Krueger look at changes in employment and find a small, positive, but insignificant effect. The conclusion is that the minimum wage increase did not cause unemployment. Politicians frequently cite the paper to draw the much broader conclusion that minimum wage increases don’t, more generally, cause unemployment.

It isn’t a persuasive analysis. It’s further confirmation to the believer, but it won’t convert anyone.

Suppose that, prior to the analysis, I disagree with the idea that labor demand is flat or upward-sloping. Now, after the analysis, you tell me: either you have to believe that demand is flat or upward-sloping, or you can believe that these New Jersey and Pennsylvania counties are not really comparable.

Of course, I’m going to conclude that these counties aren’t really comparable. If I have to choose between “People don’t care how much they pay for things” and “These counties aren’t comparable in some subtle way. Maybe another policy or shock that happened at around the same time wasn’t distributed evenly across New Jersey and Pennsylvania counties. Maybe these counties are just wholly incomparable because of a long history of differences in policy and polity between the two states that change how they respond to various shocks. Maybe a silly shock that we’ll never understand happened. Maybe the unemployment effect is delayed because of adjustment frictions.” All of those explanations are more credible and convincing to me than “people don’t care about how much they pay for things.”

So, the study fails to convince anyone. Proponents of a higher minimum wage like the study, but they don’t need to be convinced by it. Sometimes academic economists who find themselves in policy roles where they need to pretend to like minimum wages might find it useful as air cover to explain away past writing that says minimum wages cause unemployment. “Oh, I used to think minimum wages were bad, but then I read Card and Kreuger.” Hot take: No true believer in downward-sloping demand curves has ever been convinced by this. It’s so much easier to conclude that parallel trends don’t really hold or the post period isn’t long enough to see the effect than to invalidate a simple, powerful economic theory on the basis of a diff-in-diff.

This isn’t to argue about an economics paper from more than thirty years ago. It’s to demonstrate that:

Shocking results need mundane assumptions to convert new folks to your way of thinking.

We should be credibility-maxing?

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

*This point isn’t about Bayes’ rule. I understand how my joint prior over the assumptions and conclusions might cause my posterior beliefs about the conclusion to change, even if I view the assumption as less credible than the conclusion in terms of my prior’s marginal distributions. It’s an argument about how people are actually convinced, i.e., I don’t have a tightly constructed joint prior distribution over the likelihood of New Jersey and Pennsylvania counties being comparable and the slope of the labor demand curve. I weigh how plausible the assumptions are and how plausible I think the conclusion is, and that determines whether I end up convinced by the analysis to change my mind.

Ever since I decided to blog well, I’ve been trying to figure out the right way to write a good “links” article. All the cool kids on Substack do it, and I read the Internet. I told myself I ought to be able to do it too. Two statistics blogs and one links blog a week sounded plausible.

Like most things, though, links blogs looked so easy to get right until I tried to do it. So, we’re experimenting with formats. I think I like this one. Two Good Things. Two stories or clips that I think our worth consuming this week. We’ll get there eventually.

This one has bathhouses and AI skeptics.

Subscribe now

Bathhouses

Joe Hovde has a great blog on this company called Bathhouse. It’s a public sauna trying to become a third place—presumably, for the kinds of people who feel comfortable less-than-clothed around strangers: hot people.

A little trend. I’ve noticed that proposed Third Places end up getting accused of being cults—which is natural. They’re trying to replace the Church or the Bar. A Third Place has to have aspects of religion or addiction to satisfy the human need for connection via transcendence or dependency.

I don’t see it crossing generations, but fingers-crossed for them. That’s the key to a successful Third Place. There’s always the old guy at the Bar and the kids with fake ID’s. Church has Sunday school and funerals. Even coffee shops—when those were a bigger deal—cross generations. At least the marketing makes it look like a young single thing?

The website has an “AI Search” feature.

In any case, the article’s a great dive into the business and a category I didn’t know existed before. Check it out.

[I dig this blog in general, by the way. It’s always got some interesting dive into an industry I either don’t know much about or don’t know enough about. Good place to learn stuff.]

“[LLM Companies] Shouldn’t Be Allowed to IPO”

I thought this segment on Bloomberg with Ed Zitron where he hates on AI and AI companies was entertaining, if not persuasive.

Mr. Zitron doesn’t like AI or AI companies, but he has much stronger arguments for the Company’s IPO prospects than the Product and he kind of mixes the two willy-nilly.

The part he’s probably right about is that no one has a great business model yet for how to make money by supplying LLMs (or maybe the market structure just isn’t right yet to do it: you can’t get enough of a margin), and the set of tasks LLMs are good at is much smaller than the AI-positive discourse admits.

Folks are allowed/encouraged/threatened to use AI, and, for many tasks, the price of the tokens is still way more than the value delivered by the LLM. That’s how you get places that had never used AI until a couple years ago, burning millions of dollars to let folks spin up questionable (but expensive) dashboards without knowing SQL or save some Engineer a few (very expensive) hours.

The part he’s (super?) wrong about is this whole idea he often puts out there that the LLM product itself is bad-to-useless. There are nuanced takes on how LLMs can reduce productivity (by yours truly) because workers will substitute towards the LLMs and away from effort even for tasks where effort is more productive than LLMs, but there are other tasks where LLMs absolutely deliver strong productivity gains.

I do not want to go back to debugging without LLMs.

I spent the last weekend debugging this side project (a Rails app), and it would have taken much longer and been so much worse if it had just been me searching for bugs, writing tests, etc, instead of using Codex. I think I spent around 20 hours debugging the app with Codex (i.e. not just prompting “fix all bugs, make no mistakes” like you hear from some portions of the internet…). It easily would have taken me a few weeks without it, and it probably wouldn’t have been done to the same quality because I’m human, and I miss stuff (edge-cases, various long strings of scenarios that cause odd state X, etc).

The LLMs are a little like chess computers when it comes to debugging. Humans miss possible moves in chess all the time—even grandmasters miss mates-in-one in short time control games. Finding all possible one-shot moves on an 8x8 grid is a trivial problem for a computer, though. Similarly, LLMs can come up with ways in which someone could get something funky by hitting a certain string of endpoints much better than my feeble, meatsack mind.

That being said, it is definitely true, and this is Ed’s main point: the value of debugging code, formatting docs, etcetera does not justify the stratospheric level of capital and debt being loaded into these AI Labs. They kind of have to create God or at least replace Raytheon.

I might like (very much) being able to type “replace all my poorly-formatted math in this doc with well-formatted math,” but I’m not willing to pay much for it. I doubt it drives significant company revenue. Or, for example, with the above experience, debugging a web app:

I did it all in Codex (Free).

Yeah, so it was free for me. It wasn’t close to free for OpenAI, I’m sure. How much would I have paid? Not much. It’s just a little side project. If it was a major potential revenue stream? I guess I’d pay a thou or so for access to debugging for a month or something? But I doubt I’d have to. Competition is still too intense. I’d just substitute to Gemini or Claude. So prices stay low…

Until the shakeout stage of the industry (start here for industry lifecycle literature).

I think two things are true:

1. LLMs are super valuable.

2. LLMs are over-valued.

But they won’t both be true when the market for Anthropic’s future dividends becomes liquid and well-informed…

I hope the S-1 becomes public in time for beach reading.

Yes, we do have beaches. Lake Michigan counts.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

A/B tests don’t have to be Experiments. You can use them for other purposes. Corporate political propaganda, for example. Like when politicians cite random studies they haven’t read on a “debate” stage, the point isn’t the study itself. It’s the conclusion.

No judgment. Politics has its place.

The difference between an Experiment and Propaganda is whether you want to learn something or convince someone.

Subscribe now

If you want to convince someone, you will choose metrics that are all but guaranteed to go in a certain direction.

For example, you might remove a screen from a checkout flow and make your metric “time spent in checkout.” Of course, the time spent will go down. You already knew that before running the experiment. You’re building up a case for your preferred product direction, the one that hits your OKR. Not running an Experiment.

If you were running an Experiment, you’d want to know: what do we lose by dropping that screen? What do we gain by dropping it? What metric can I develop that lets both the positives and negatives of dropping the screen shine through, their relative light depending only on how customers behave? Maybe the extra information from the extra screen is worth nothing. Maybe it is everything. The choice of metric should not assume either. It should allow for both cases and let customer behavior decide.

In other words: Experiment metrics maximize surprise. Propaganda metrics minimize it.

Suppose you want to learn something. How do you go about finding a metric?

In the screen-drop example… Suppose the extra screen asks the customer for a bit more information about why they purchased the product. Maybe that information feeds later personalization or targeted messages to get the customer to buy again or subscribe or otherwise take the next step in their journey. It’ll always be something like this. It’s not like the extra-screen was just thrown in there willy-nilly. The folks who added it did so for a reason. We should know why before we knock it down.

We don’t want to use the probability of making it to the end of checkout because, of course, taking away extra screens helps with that.

We also don’t want the metric to be a conversion rate of users to taking the next step. Of course that’s going to fall. The extra screen makes the ad-targeting better.

What we want is a metric that could go either way.

For example, the joint probability of both completing the checkout and taking the next step—not the conditional probability.

If the conditional probability of taking the next step falls, but the number of customers who take the next step increases, then dropping the screen makes sense. You’re improving the probability of checking out without dropping the probability that a top-of-funnel visitor gets to the final stage of the journeys, even though targeting gets worse.

If these two steps have different economic models associated with them (maybe the checkout is the Basic subscription and the extra-screen provides information used to market Premium to the user), then make the metric take that into account. Give each one a dollar value and try to maximize expected dollars.

The point is to get the tradeoff in there. We’re making a move that increases the pr(checkout) and decreases the pr(premium | checkout). So we want to know whether pr(checkout, premium) = pr(checkout) x pr(premium|checkout) increases. We’re surprised either way because, while we know pr(checkout) will increase and pr(premium | checkout) will decrease, we don’t know the net effect.

So, we’ve set up our experiment metrics to learn something, and, along the way, we’ve figured out exactly what we’re uncertain about on the product-side. That’s the right way to select metrics. Like most problems, it can be solved by the algorithm:

1. Write down we know.

2. Write down what we don’t know.

3. Experiment to bridge the gap.

When I was but a wee lad learning to shoot, my dad told me the trick was to “Let it surprise you.” If we anticipate what’ll happen when we pull the trigger, we won’t hit the target.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Production functions matter. They describe how firms or people turn raw inputs into outputs, i.e., technology. They constrain what’s possible, like demand determines what’s preferable.

Subscribe now

Most empirical work on production functions in economics looks at manufacturing, agriculture, or resource production because their inputs and outputs are easier to measure. But knowledge work is the first job class to be altered by the LLMs. Codex isn’t building bridges anytime soon. So, modeling how knowledge workers produce output and how LLMs affect production matters, but I don’t have any data to work with...

We can learn a lot from theory, though…

LLMs take context (human knowledge) and use it to generate output effortlessly. Without LLMs, humans would still provide their knowledge input, but they also need to supply elbow grease: effort. This blog will show that uncertainty about the best way to do things—and the option to apply effort to counteract that uncertainty—drives how LLMs influence technology choice, production, and the returns to more accurate beliefs.

The plan is for this to be a multi-part blog series with different models of how LLMs transform human context into output.

This post treats LLMs as pure automation machines. They take instructions from a human and act just like a human would, but they save the human significant effort.

The key part of the theory is the role of “uncertainty” in how best to do tasks. It turns out to be critical for understanding the role of LLMs in knowledge production work. We’ll show the following about how uncertainty changes the role of LLMs:

• For low-uncertainty tasks, workers will use LLMs, which will reduce the time it takes to complete a task or improve the quality of their output.

• For moderate-uncertainty tasks, workers will use LLMs, which will increase the time it takes to complete a task or reduce the quality of their output. The worker’s preferred technology will not produce the best output.

• For high uncertainty tasks, workers will NOT use LLMs. Instead, they will use more effort to counteract their uncertainty.

Production without LLMs

Suppose a task can be completed in a certain amount of time, determined (without LLMs) by how smartly people decide to do the task, and the amount of effort they put in to get it done.

[You can think of “time” more abstractly as the inverse of “quality”, where lower time is equivalent to higher quality output, etc. But I’m going to stick to the strict time metaphor for ease of following the model.]

s is the best way to do a task, but it is unknown. Knowledge workers have beliefs (knowledge) about how best to do things, given by:

Without LLMs, the time it takes a knowledge worker to do a given task is given by:

Where z is the way the knowledge worker decides to do things, and e is the amount of effort they put in.

Knowledge workers care about doing a good job, but they also don’t like to put in effort. They choose their effort and the way they want to do the task to solve this problem:

The expectation is with respect to the knowledge worker’s subjective knowledge of the best way to do things. Because the expectation minimizes the square loss, the best z will always be m. By definition, E[(s-m)²] = v. Because this problem is convex for positive e, we can solve for the effort level from the first order conditions:

So, at a high level, uncertainty defines effort. The more uncertain we are, the more effort humans supply to guard against unexpectedly large errors in the course of action they decide to take.

Therefore, the (objective) time it takes a given worker to do a task without an LLM is:

LLMs as pure automation machines

LLMs work by receiving context from humans. Humans can transmit their beliefs to LLMs as context, and then the LLM decides what to do with it, i.e., the LLM production technology is:

Where a gives the equivalent effort level of the LLM’s output. It measures the LLM’s accuracy or performance and might be captured directly by how long it takes to coach the LLM on exactly what you want it to do or on the quality of the output it produces.

Our first model of LLM production is what I’ll call “the pure automation model.” In this model, the LLM performs the task in the same way a knowledge worker would, but effortlessly. It doesn’t have its own beliefs. You tell it what you want done, and it does it. So: f(m,v) = m, because you’ll choose to tell it to minimize the expected time given your beliefs, and the (objective) time it takes to complete the task with an LLM is:

It’s a simple model, but it actually has some semi-non-obvious predictions.

Workers choose to use LLMs when they are MORE certain of the best way to do things

Workers choose to use LLMs-as-automation when they are more certain about how to do things.

The result follows from effort substituting for uncertainty. Workers can put in more effort to mitigate their uncertainty when using the non-LLM technology, but the LLM technology doesn’t respond to their effort.

So, workers choose to use LLMs-as-automation whenever their uncertainty about the best way to do things is low enough:

So, with the introduction of LLMs, only the more ambiguous tasks will be done by the effort technology, and, as the effectiveness of LLMs (a) increases, more and more ambiguous tasks will be done via LLM.

The time to complete tasks increases for some workers when they have the LLM option

This result is “obvious” when you write down a model like this, but not clear at all without actually writing down the math. The key idea is simple, though:

Workers want to avoid spending effort, so they will choose to use LLMs even if the effort technology is faster than the LLM technology.

We already have the rule for when workers will choose to use LLMs. So, to characterize when LLMs increase the time a task takes, we need to look at the conditions under which LLMs are slower:

So, for v = 2a³, the LLM method of production is slower, but workers choose to use it (because 2a³ < 8a³); otherwise, they have to pay the effort cost. Folks don’t just try to minimize time (equivalently, maximize output quality).

For a moderate level of uncertainty, i.e., for a³ < v < 8a³, knowledge workers choose to use LLMs despite the longer task times (equivalently, lower-quality output) they entail.

This production model is static, but it implies something about dynamics. If AI quality (a) improves only slowly, some industries stall in this middle ground, where workers choose to use the LLM technology, but it gives worse/less output. Firms using the effort technology outcompete them. So, AI adoption is quick in industries defined by moderate ambiguity about how best to do things for firms that make it readily available to their workforce. AI adopters lose out to the Reactionaries: firms that force workers to use the effort technology and refuse to pay Anthropic or OpenAI.

Automation LLMs decrease the returns to accuracy if uncertainty is low

Do more skilled workers do better or worse with the option to use LLMs? Do LLMs compress the productivity distribution, allowing less-skilled (maybe early-career) workers to catch up to more skilled workers—or do they widen the gap between the skilled and the less-skilled?

Suppose everyone has the same level of uncertainty (the same v). Then we can define more skilled workers as those with a smaller d = (m - s)².

Shrinking d decreases time by v^(-1/3) for the effort technology and by a^-1 for the LLM technology. So the reduction in time from more accurate prediction is greater for LLMs if:

This is the same as the rule for when LLMs are slower. So, for highly uncertain tasks, the return to skill is greater for the LLM technology than for the effort technology. For low-uncertainty tasks, the return to skill is greater for the effort technology.

If we incorporate the choice constraint into this, we can partition the results like so:

Without LLMs, the return to skill would always be v^-1/3. LLMs increase the return to skill for moderate uncertainty but do not affect it for high-uncertainty tasks (because workers choose effort over LLMs). For low-uncertainty tasks, LLMs are faster than the effort technology and reduce the return to skill.

So, moderate-ambiguity tasks will demand more skilled workers with LLM than they did with the effort technology. Less ambiguous tasks will demand less skilled workers.

So…

If we can compensate for our uncertainty about how best to do something by increasing effort, then we will only do tasks with low uncertainty via the LLM technology. Skill will matter less for low uncertainty tasks. And tasks with moderate uncertainty will now take longer (or be lower quality) than before (tell me you haven’t felt this at some point at work since LLMs came into their own) while demanding higher-skilled workers because of the increased return to skill.

In this model of LLM production, the knowledge worker provides a vital input: their own beliefs about how best to do things. LLMs don’t work without the worker. The model helps us isolate the effects of the automation component of LLMs, but, of course, LLMs do more than automate production (although for some applications… that is pretty close to their purpose).

In the next model of LLM production, we’ll allow the LLM to have beliefs of its own about how best to do things and open up the decision of whether to use a worker at all…

Math…

Although this model of production is relatively simple, it makes arguments that get bandied about in the AI discourse precise and concrete, with at least a plausible theory for what gets done by AI and what doesn’t. Writing down exactly what you are assuming about the structure of folks’ decision-making process clarifies. Implications naturally follow from assumptions and structure.

You can do more than just say things.

Saying, “I think LLMs are great for generating quick queries or searching customer calls, but they’ve made docs less useful because they’re all AI-generated and very hard to parse,” is all well and good, but it’s just like your opinion, man. With a clear model, it starts to make sense.

Writing a good doc is a great example of a moderately uncertain task. It’s not untrod ground, but neither is it exactly obvious how to propose a new idea effectively in a way that accurately communicates what you want to do, why it’s a good idea, and what might go wrong. So, the degradation in quality with the LLM technology available makes sense. It follows from folks avoiding the effort cost to work on a moderately ambiguous problem.

On the other hand, a quick query or labeling customer calls is something you know exactly how to do. It would just take you a relatively long time to do it yourself. And the model says that low-uncertainty scenarios will be better with AI because you can get a baseline effort level without workers having to pay the effort cost for the exact kind of task where the returns to effort (from the worker’s perspective) are low.

The advantage of writing things down precisely in math language is that it makes it clear what assumptions drive your thinking, and it creates natural implications of those assumptions that you’ve not thought of before.

Writing, in both Human Language and Mathematics, is how humans think. Computers have RAM. We’ve got Emacs, Google Docs, 3x5 notebooks, and cave walls.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

We’ve got a few stories from the SaaS wars—and takes, of course. We have an infinite supply of takes, far exceeding demand.

Subscribe now

• Dispatch from the SaaS data access wars:
  Every week, there’s a new story about so-and-so blocking data access from such-and-such. This week, Microsoft is supposedly blocking a certain kind of access from external tools to Power BI. The article contains this haunting quote: “The friction centers on Power BI, a Microsoft product nearly all Fortune 500 firms use to analyze data about their operations in charts and other visual formats.” Really? Nearly all?
  
  I’d like to take this chance to share a conspiracy theory. Power BI isn’t real. I hear about it all the time. It’s allegedly everywhere. And yet… I’ve never used it or seen it be used. I’m tech-old. I’ve been around lots of data environments. Sounds fake to me.

• Did you hear SpaceX is going to be IPO?
  No? I didn’t either. I couldn’t find a single article about it to share. See if you can find one somewhere on the internet.

• Customers demand shorter contracts over uncertainty of “what comes next”
  There’s a story on AI dollars crowding out traditional business software purchases. The pool of dollars isn’t fixed, and if there is a large return to these AI products, why couldn’t these firms borrow to cover token purchases and repay the loans with excess profits? So, something is wrong here. Either credit markets (unlikely), the returns from AI spend are currently negative (plausible until companies learn the right way to deploy it at their firm), or there’s a principal-agent problem (highly likely). The person doing the buying has a fixed budget, so even though the company could just expand borrowing, there’s no cheap way for the agent to pass the information up the chain.
  
  The second half of the story is more plausible, but speculative. Companies are worried that a wave of new, better products will emerge over the next few years and are reluctant to be locked into a long-term contract. That makes sense, but I really haven’t seen any evidence of useful AI-native business software, aside from the models and coding stuff coming from the primary labs. The AI components added to existing SaaS have been useful, but AI features are not really useful on their own. You need something that works well for all the more mundane features of the app first, and that turns out to be a hard problem.
  
  It reminds me of every time I’ve seen what I think is an opportunity to exploit in an industry. I realize that, while I might be right about the opportunity, to actually profit from it, you’d have to launch a competitor that does all the other aspects of the industry right and then just adds the new insight. That’s very hard to do.

• Intuit moves to consumption-pricing.
  Everyone is doing it because it’s the only thing you can do, now that the Variable Cost Revolution is well underway. What is the Variable Cost Revolution, you ask? For a long time, one reason software companies were so profitable was that their costs were almost entirely fixed. Costs did not increase very much as they added more customers. So, with scale, they made oodles of money. But AI has changed that. If you run an AI feature in your software, you have a variable cost product. As demand for the product increases, your costs increase meaningfully. LLM-based features have changed the cost structure of the entire industry, and the pricing structure follows the cost structure.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn