Subscribe now

The first rule of a good model is that it’s explicit. It goes step-by-step.

Isn’t a model. It doesn’t say anything. It doesn’t tell me how the world works. Good models that change people’s minds explain things. They tell us how the whole world works, step by step, tracing the path from point A to point B. The players have incentives, goals, and structures they have to operate within.

Simple equations, although usually more robust (e.g., regressions, treatment effects) in the sense that they are true in more or more plausible states of the world, lose to models because they purchase their credibility at the cost of failing to provide meaning—“it made revenue go up” not “customers who value feature X have weaker price elasticities and…”—and we want to understand.

For example, we might compare two price vectors p and p’ in an ideal experiment. We can pick which one is better by comparing, say, average revenue, but you’ll be surprised how unconvincing that is unless there is some model for why p’ ought to do better than p. If you can say p’ takes into account the substitution patterns between the products you’re selling, while p prices each product as if it were independent, you’ll get immediate buy-in and even a thought that we should apply this everywhere, to other product areas, etc. If it’s just… empirically better, we’ll ship the treatment, but we’ll be wondering why constantly. If things go south, doubt will creep in, and we’ll start to suspect it was a fluke.

This is why religion crushes simple non-belief in the marketplace. It explains. It tells us how the world works, a real model of things. Non-belief doesn’t do that. You have to add to it to get a model of the world. Credibility doesn’t stand a chance against a model. No one’s overthrowing the bourgeoisie because they don’t believe in God. They have to believe in something else first. The dialectic. It’s important to consider these things when designing models to pick the optimal promo depth for your e-mail campaign.

The point is that building a model of what you’re doing matters. Customers choose among these alternatives. They value such-and-such. And, look, if that’s true, then the right price is $53.92 because… etc, etc.

So many good insights fall out of models. You can figure some things out deductively. The data is limited, and if you make it solve everything, it loses the power to tell you the answers to the truly unknown questions. For an extreme example that’s hopefully illustrative:

Suppose you have two products:

1. You can read an unlimited number of books from a certain library, and the reading app’s interface is Gold.

2. You can read two books per month from the same library, and the reading app’s interface is Blue.

Look, the product’s got two features, and some people might like Blue and other people might like Gold. So we’ve got both horizontal and vertical differentiation. Who’s to say which product’s price should be higher?

But we know, right? We have a model of the world that says readers care more about the number of books they can read than the color of the UI. We have another model of the world that says, in fact, we should just let people choose the color. No one’s willing to pay anything for that.

We could test this, and statistical noise would give us various customer valuations for Blue and Gold. It really would. We’d start talking about how people value Gold at around $2.31/month. There’d be confidence intervals. It’d look like analysis. We could make plans to drive Gold upsells.

Or: we can admit we know things, and if we use statistics to try to learn what we already know, we’ll just be drawing random shocks and tap-dancing around the truth.

The thing we don’t know here is how much people value reading two books per month versus the unlimited option. The signal we get from the data here is worth the noise. We can’t just know this from having a basic model of the world.

If instead of two books, it was a million, then we’d be back in the world of modeling again. No one could physically read that many books in a month. We don’t need to test. We have a model of the world—linear time—that collapses these two products into one.

When you get more explicit with even very simple modeling, surprising things fall out. Like, we have a suspicion that it takes us longer to use the Robots than it used to take us to do some tasks, but we use them anyway. Why? Well, it turns out a very simple production theory makes exactly that prediction…

We’ve got limited time, limited data, and infinite questions. We have to choose what to ask. Models help us decide what’s worth asking and what’s not.

Thank you for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Subscribe now

A take with no equations? Sounds risky, but let’s give it a shot…

In the ancient past, biz writers read their documents more than all other readers combined. LLMs changed that. Now, the Reddit Regressions frequently generate memos, design docs, PRDs, etc, etc with only light editing by the responsible parties.

I don’t like it.

1. All Change Is Bad, and the kids really must get off my lawn.

2. Biz writing is not about the document itself. A newspaper article or a book is its own end, but that’s not really the case for the memo. The point of writing it out line-by-line is to think through the problem and its solution step-by-step. Maybe other people will read it and be convinced, but primarily, the author organizes their thoughts, folks add a few comments, and then it serves as the basis—an outline—for a live discussion where the decision actually gets made.

The traditional biz doc represents, if not an engrossing read, how a person thought through a problem. It might lack all the other elements of “good writing,” but it has the most important one: the author wrote it to say something, and they thought it through.

The LLMs make every doc, wiki, PRD, etc, read like a high school essay on The Scarlett Letter—naturally, because they are written for the same reason: a box must be checked. The main difference is that the pre-LLM high school essay usually has an endearing personality and a visible human struggle to make word count: “I think it is bad what they did to Hester Prynne, really. I certainly wouldn’t like it were it done to me, the Golden Rule and all…” The LLMs have that repetitive, exhausting quality sans voice.

Writing it out is the best way to convince yourself that the idea you had really works, when you consider all the information carefully. When drafting, you end up thinking things like, “Oh, wait, I forgot about the subscribers who are still grandfathered into the 2007 plan.” Writing moves information from RAM to Disk, freeing up working memory to analyze your 2-am-brilliance critically.

“Write an argument for…” or “Write a PRD for a new feature that does X, Y, Z” are silly prompts because if you haven’t written it out, you haven’t convinced yourself the argument or plan is true or a good idea.

“Tell me why this rough doc is wrong…” is a much better prompt. The Robots have good thoughts once they are dealing with real output, and are great at catching questions you forgot to ask.

They’re terrible at generating biz docs from a paragraph and data—which isn’t really a knock on ‘em. They got a paragraph prompt and had to write a whole essay, and their topic isn’t an important 19th-century novel, but the packaging of an HR SaaS product.

It’s a tough assignment.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

There’s an inverted U-shaped relationship between competition and innovation.

With no competition (including no serious threat of entry), there’s little incentive to innovate because you just extract monopoly rents, and there’s little extra profit to make by making a better product, past a certain quality level.

Under perfect (intense) competition, any product improvement gets competed away, driving profits back down to zero, so you get no returns on the investment you made to improve the product.

This is the justification for patents in, say, drugs. Any new drug can be cloned cheaply, creating intense competition post-innovation, so there’s no reward for investing all that dough in developing the drug. So, we reduce competition with patents.

You need just the right amount of competition to get strong innovation. Not too much. Not too little.

Subscribe now

I wonder if this isn’t an argument that the LLMs will end up getting government money.

It sounds silly—and I tend to be reflexively against this kind of industrial policy—but there’s essentially zero property rights. No lasting product differentiation. The lagging models catch up to the frontier models in a matter of months, which is not material for current use cases. They produce (significant) economic value, of course, but I think there’s a real doubt the LLM producers can capture it, given the market structure.

For now, folks are still betting that there will be a winner of the race. But if this market structure sticks around too long, without any shakeout, investors won’t continue to believe that. They’ll start to believe there are no profits out there to offset the scale of the investment.

There is some return to packaging, software, harnesses you might bundle with the LLM. I might like the ergonomics of Claude or Codex more, but that’s small-fry differentiation compared to actual model performance. I can’t imagine the returns to this kind of product differentiation are enough to encourage the investment needed to make Silicon Gods.

When products are valuable, but the ex-post profits will be competed away, we tend to do… something. Patents or other restrictions on entry are the most common approaches, but they won’t work here—at least not so far as the US Government is concerned. China will keep producing these models, economic returns be damned. The Chinese models can’t be easily excluded from valuable markets, and the models are valuable…

And so, if someone can’t break away and competitive profits are all that await Anthropic and OpenAI, I’m betting we’ll end up subsidizing investors to encourage future investment. Yes, it’ll make this whole corner of the industry a little more… mucky with the usual rent-seeking, and the incentive problem of encouraging riskier investments, but I could see the government reasonably deciding it’s less bad than the alternative.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Subscribe now

Award-winning directors and actresses alike need business advice from esteemed corporate drones, like me. I wish it weren’t the case, but it is. Both of these ladies have failed to fully and completely cash in on remake/sequel/prequel/branding fever, and they have such a golden opportunity.

Lost in Translation 2.

One might think a sequel is unnecessary and, in fact, destroys the very arc of the movie, the ambiguity, the aftertaste. It is not that you are wrong. You are worse than wrong. You are not asking the right question. The right question is: will this shit sell? The answer is: Yes. With the right script. With the right direction. With the right actress reprising her role.

The people cry out for it.

Look, I’ve even sketched the plot for you. Picture this, Sofia.

I walk into your office, a bundle of nerves, my harried look a dead giveaway that I’m here to pitch you. You’ve heard it before.

“I have a pitch for a movie,” I say. You roll your eyes.

“It's Scarlett Johansen playing the older version of her character from Lost in Translation. She didn't know what to do after Japan, so she went to law school. She went to Yale after all, majored in philosophy, thought she could be a writer. Where else does this type end up? It's only natural. She has to do something, and there's law school. Something.”

You lean forward in your chair, intrigued. Finally, someone else saw what you saw…

“She's working a case, a morally questionable one. I don’t know. She’s defending clubbing baby seals in the Arctic. But it doesn't start there. It starts with her defending something more innocent, but still guilty. And then it's just a little more each day. She meets a guy in private equity working on an acquisition. It’s New York, okay? She’s divorced that photography guy because he didn’t support her going to law school. ‘I just want to make something of myself,’ she said. ‘So does everyone. But a lawyer?’ He didn’t understand. They drifted apart. Everyone's mad at the new (much hotter) private equity guy because he and his firm laid off three entire Midwestern towns stocked with the most sympathetic people. Scarlett and hot finance guy bond over being hated and fall in love, and then the rest of it is a rom-com where they tell jokes about clubbing baby seals and firing people while dodging protestors and having red paint thrown at them. It's really not their fault. Why shouldn't they be in love? It turns out because they are conflicted on an upcoming acquisition. They are on opposite sides. She’s representing another bidder for this distressed Midwestern amusement park company, and they lose touch because they can't communicate until the trial. And then she remembers that she wants to be a writer, so she writes a romance novel about it, working title Conflicted, and she tries to get it published, but no one wants it because it doesn’t ‘speak to the market’.”

“Dear God,” you say, evidently impressed. “Please leave.”

You need a moment to collect your thoughts to respond to my artistic brilliance. I understand, but I offer one more nugget to convince you.

“The movie ends with a flashback to the first installment of the Lost In Translation universe. The part where Bill Murray whispers in her ear. You can hear the audio clearly, and Bill Murray says, ‘Look, I’ve done art. I’ve been an actor. Now look at me. I’m reading whisky ads in Japan. Just take the LSAT, kid. Just take the LSAT.’ And they kiss.”

“Security,” you say.

Thank you for reading, Ms. Coppola and Ms. Johansson, and for your consideration,

Zach

(I am currently on Day 1 of 2 of unemployment. I should not be left with nothing to do.)

Subscribe now

Dashboards are the worst form of self-service analysis, except for all the other forms that have been tried from time to time...

They are terrible because they are inflexible. Dashboards take material time to build, yet they don’t answer 85% of the class of questions they were supposed to answer. The questions are ever-changing, while the dashboards are static, fixed. They need another filter. Another cut. Another unit. Another metric. So, what becomes of the dashboard? Reactively, we expand its filters and parameters. Still, it sits there unused, and folks continue to ping their friendly neighborhood analysis person to do whatever needs doing.

And then: what does the analysis person do? They don’t send back a dashboard-like document, a sentence of text around charts and tables. No. They write a memo [a “doc”], describing what they found and what the analysis means.

The point of dashboards is to enable self-serve analysis, yet they look nothing like the analyses data scientists produce. Think about how weird it would be if a data scientist just sent you back a couple of graphs when you asked them to look into something. There should be a narrative, an argument, facts… Something.

Why should self-serve analysis look different than the analysis I do?

The only difference between self-serve analysis and bespoke, artisanal, handcrafted analysis from the Analyses region of Northern Illinois should be who ran the thing.

Self-serve analysis has a great premise. The business questions come from one set of humans, but the answers traditionally require another set of humans, who (on occasion) have to eat, sleep, and play Cyberpunk 2077. What if we could scale the answers? What if we could write the analysis once and scale it so that we could automagically generate answers to a broad class of questions? If only Eve hadn’t….

So, this is my hope for the BI Tool of the Future, and, because I’m an Optimist, my hopes are the same as my predictions: we need a Memo-Generating Machine.

A Memo-Generating Machine does the following:

1. Within a certain class of questions, it translates the details of a questioner’s ask into code that returns an answer.

2. The answer is a memo, explaining how we answered the question, why we did it that way, and what the analysis shows.

I’d been toying with how to make the mystic machine real for a while, and then, suddenly, it became commonplace. In 2019-2022, I wrote a bit about this version of BI, and made a Rails app that implemented a rough version of the idea. It worked okay, but you had to do a decent amount of the memo generation yourself as the analyst, which made it difficult to automate for more open-ended analyses. So, it didn’t really solve the problem very well. But things have changed since those dark days…

You’ll notice, of course, that the Memo-Generating Machine is more or less Claude Cowork with well-specified instructions.

Claude doesn’t generate dashboards when you ask it to find the answer to some question you have. It writes memos, the standard output of data analysis, like a good little robot.

So, I dig what LLMs have brought to the data analysis scene because, despite their flaws, they have at least identified the correct form of the output, which is the most important thing.

The one downside of LLMs is that they’ve enabled a lot of truly shitty data analysis, and, in typical tech fashion, they have done so at scale.

The problem is they do not put structure on the form of the analysis. Often, they get some context on the database structure and how to run queries, and then they just kind of react to prompts, generating queries—often incorrectly—in subtle, difficult-to-detect ways, or they simply answer the wrong question. A lot of the folks who want to run self-service analyses can’t really tell when the LLM does something wrong or silly because they need more context about the database or statistics, etc, so it’s not really a self-service structure on its own.

You need a lot of context to do a strong analysis, and it’s unlikely to be provided by a generic solution, like a metadata layer on top of a database, because it’s not just the data that matters but how to analyze it.

So, a raw conversation with an LLM merely backloads the usual ask to the analysis team. Instead of “can you pull this?” the question becomes “can you check this?” Which is an improvement, but it’s not really self-service.

The good news is that it doesn’t have to be this way. One structure I’ve been experimenting with is being heavily prescriptive about the form of the analysis but lightly prescriptive about the form of the inputs.

Something like the following structure for, say, a Claude skill:

1. Tell the little robot to extract some info from the user’s prompt…

2. Tell it to run this exact script to do the analysis, filling in the parameter gaps as needed from the prompt. Here, I insert an exact R/Python/SQL/Shell script that does whatever analysis I’d do for this class of question directly into the skill.

3. Tell it to write a report on the analysis, focusing on …, noting …, recognizing that X is true, and giving it instructions for interpreting things, etc.

4. Feed it some of my artisanal, handcrafted memos and tell it to write like me, a human, not like it, a robot.

Step 4 is still a very much unsolved part of this version of self-serve analysis. Steps 1-3, the status quo robot does reasonably well. It does a great job of extracting parameters from natural-language input and asking follow-up questions if the user didn’t include some relevant info. It always follows instructions to run the exact script. It mostly does a good job of including the relevant constraints on the memo, but oh boy, does LLM writing suck.

It’s horrible. Worse than lifeless. I wish it were what we used to call “robotic.” Unnatural, overly stiff writing would be so much preferable to its imitation of a high school book report. Insufferable adjectives. Why does it insert imagery, metaphor, and simile in contexts where they make no sense? And so much of it. So much of it. They write, and they write. I think it’s because Anthropic charges by the word.

Things haven’t changed since Dickens*. Everyone responds to incentives. Great writers and robots alike.

* Dickens was not actually paid by the word. His tokens were “installments,” which had to be roughly 32 pages. Still, naturally, he wrote to generate installments.

AI-generated text exhausts the reader.

I have faith we’ll fix this, and, in the meantime, this version of the Memo-Generating Machine works. Highly recommended.

Aside from the writing in the memo itself, the other problem to solve with the current Memo-Generating Machine is cost. Running an analysis now has a variable cost beyond the querying and compute costs, and it can add up. Dashboards do have a significant advantage here because they’ll usually materialize a table once and run various, relatively cheap transformations on this dataset. The Memo-Generating Machine does much more analysis on the fly, so it’s more expensive (but more flexible), even before accounting for token costs.

That said, the more tightly constrained analysis setup above greatly reduces costs compared to having the LLM create its own queries or whatever. They tend to use a lot of compute, process tons of data, and burn tokens while the robot backs into something approaching a reasonable analysis via an expensive iteration process.

That’s basically what I’ve found works for cost control. If a skill is too expensive, constrain it and allow less discretion.

The “script-constrained” LLM doesn’t use nearly as many tokens, though, as the standard vague prompt to an LLM with access to your data warehouse, so it’s dramatically cheaper than that increasingly popular option.

Anyway, this is where I think BI tools are headed. Away from dashboards and towards memos. Which was always the right direction for things to go. So… good news! There are still problems to solve to make it work swell, but I’d guess this structure is the standard form of self-service analysis in the next five years.

/remind-me 2031-07-10

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Tesla caps per-employee AI spend at $200.

Tesla—and other companies—have started limiting AI use at the employee level. The theory behind capping individual users—which I buy—must be that each additional dollar an employee spends on LLMs generates less value for the biz, so limits should help prevent employees from wasting the budget on low-value actions that cost more than the value they create.

They’ve “clearly” read my blog post showing that, left to their own devices, workers use LLMs even for tasks they could do better without them. For example, reformatting documents or generating fancy slide decks are two mostly unnecessary tasks that AI can handle in the background, but it’s probably not worth the high token cost.

Subscribe now

A sort of obvious LLM budget allocation strategy, if you believe labor and LLMs are complementary (which seems natural to me), is to allocate budget based on the employee’s salary. I’m sure this would go over… horribly, but it makes sense economically.

Like, let’s say total output from a certain worker with productivity parameter A who can spend up to M dollars on LLMs is:

Which, when theta is less than 1, captures the idea of decreasing returns to additional LLM spending as workers spend it on less and less valuable tasks.

Then, we can maximize net output from that worker by solving:

Which gives us:

So, we should assign a larger budget to more productive workers, and in a competitive labor market, productivity is well proxied by wages. So, we should have a regressive AI budget, with larger allocations to higher-paid workers and lower allocations to lower-paid workers.

I don’t expect many companies to do this directly even though it makes a lot of sense from an efficiency perspective. They’ll do it indirectly, of course, via allocating different amounts to different job families or levels.

It’d be cool if a company experimented with LLM budgets. Randomly give some workers more of an LLM budget and others less. Then, you’d get a decent read on the average value of tokens. It’ll be difficult to get a lot of power unless you’re a large company… like Tesla. It’d be cool to get the return on AI spend (ROAIS?). If a company does this, please publish the results. It’d be super interesting how the return on tokens (ROT… okay, this one’s unlikely to catch on) compares to their price in corporate land. The context is pretty important here: mature companies aren’t doing a ton of completely new, green-field projects. Most work is fixing or improving stuff. So, the ROT might not generalize to vibe-coding, etc, but it’d be super useful for companies figuring out what their budgets should look like.

If you’re reading this and need a way to justify doing an experiment like this to your employer, just remember that you can share the results of this experiment with OpenAI or Anthropic during the next contract negotiation and, if the ROT is low, probably convince them to give you massive discounts.

For my own personal AI spend trend… I have used LLMs less and less throughout 2026—which is sort of the opposite of how I’ve adopted most new technologies. Usually, as I move further up the learning curve, the tool becomes increasingly valuable to me, so I use it more. But it actually takes a remarkably long time to get, say, a query right with LLMs. I’ve accepted that it’s almost always easier and quicker for me to just write it, even after factoring in having to DM someone about exactly what column ABC means.

I still haven’t really found a way to use LLMs effectively for business-process-type tasks either. They generate unusable, overly dense documents and presentations with far too many images or too much formatting. Give it to folks plain, simple, and true. Write directly. Just say it. Don’t present it. You are hereby banned from using adjectives ever again. Do not editorialize. No matter what prompt you give it, it just doesn’t understand how to Be Normal (tm).

On the more positive side for the tech, I’ll never debug without AI again. It’s so incredibly strong at getting the details right when checking my code that I run basically everything I write now through the prompt “Check this for bugs. Don’t fix the bugs. Just report them.” Life-changing.

I’m not sure how well my use cases generalize, but something I’ve noticed with “generative” AI is how bad it is at generating stuff compared to evaluating stuff. If I give it a claim or some code, it’s very good at figuring out the problems from the context I’ve generated. But if I want it to make a claim or code itself, it’s a mess without significant elbow grease that I’d rather spend just writing the first draft myself and using the robots as the fanciest of spell-checkers.

I currently allocate $0 to LLMs in my personal life, aside from whatever I get from Gemini for having the “premium” Google account, which I mostly have for the extra storage, because Codex’s free tier is far too generous. I debugged this entire side project (about 10-20k LOC Rails app) easily using the Codex free tier.

I feel bad about extracting all that value from OpenAI without paying them a dime. Not bad enough to pay them, of course, but I’ll give them an in-kind payment: free pricing and packaging consulting. You should either make the free tier less generous or just kill it and offer a $5-10 plan with low limits. You’re welcome.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

The Internet is aflame with an accusation that DOGE (and, thereby, Elon Musk) is responsible for the deaths of millions because it cut funding for various aid programs. The claim comes from a study, but this post isn’t going to debate the study because it’s not relevant. I’ll just take, for the sake of argument, as given that aid cuts cause deaths and that quantitatively everything in the study checks out.

It wouldn’t surprise me if the causal story is true, but the nature of the discourse given that is bad, and it touches on interpreting policy research, so it has, however tenuous, a connection to what this blog usually studies.

Subscribe now

Every large government policy like this has mortality impacts. Every single one. If you cut taxes or raise them. If you require stricter medical ethics rules. If you criminalize drugs or make them legal. If you decide to spend X on aid, instead of 2X.

How many people did the folks who decided to fund USAID at its prior funding level kill because they didn’t fund it at twice that level?

It’s surprising that most of the defense of DOGE has so far focused on problems with the study, etc. That’s not the point. Policymakers cannot and should not choose policies that minimize worldwide mortality, and, certainly, if they do not implement such policies, they are not “responsible” for the deaths caused by not implementing the maximally mortality-reducing policy.

The Argument (a publication I generally like, for what it’s worth) says it directly in this article, right in the lede:

Elon Musk really doesn’t want you to say he’s responsible for the deaths of millions.

So yes, that the folks behind DOGE are responsible for these deaths is clearly what people want to argue in the Discourse. The phrase is probably overused, but there is a “motte-and-bailey” vibe to this where folks make the strong claim that Elon Musk is responsible for the deaths of millions and then they retreat to the claim that X people died because of aid cuts. These are different claims. Moral responsibility for death is clearly distinct from passing or not passing various laws or funding various programs.

I’m sure I could’ve saved incremental lives by donating a larger percentage of my income to some life-saving program. I’m not responsible for any deaths. That’s silly.

People die because the speed limit is 70 mph instead of 45 mph. Is every local politician who keeps the speed limit at 70 responsible for the road deaths? What about everyone opposed to the prohibition of alcohol?

It’s definitely a useful, worthwhile thing to do to study the mortality impacts of a policy. It’s part of analyzing policy impacts—alongside other impacts!—and as a study, there are no issues with this. But we shouldn’t take such analysis and use it politically to parcel out responsibility for various deaths to policymakers. It’s a never-ending war where the different sides produce excess mortality studies for each of their opponent’s policies.

Hot take: I think it’s a bad idea to make our politics one where we regularly accuse our opponents of murder if they cut funding for a program.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Today, we’re breaking from this blog’s usual topic (statistics, economics, etc) to indulge in a far more important topic: a hobby project I built seven years ago. Aside from being a trip down memory lane, it’s possible this blog has a point. Two of them, in fact:

1. Doing things just because is a perfectly good reason to do things, and

2. Extensible double-entry accounting apps are a valid form of artistic expression.

Subscribe now

It was the Fall of 2018, and I was to be wed in the Spring. I did not have much in the way of savings in those ancient days, and weddings cost money. I needed to budget for it.

Naturally, I needed to write an extensible double-entry accounting system to do this. It had to be programmable in Scheme, of course. I cannot overemphasize how much all of these requirements were unavoidable design constraints. It is impossible to budget for a wedding without Lisp.

And so, leas was born. Leas is an extensible accounting system that you can use interactively in the terminal. It’s extensible with Scheme, a full programming language, embedded in the app.

[I wrote it without thinking about LLMs—it was 2018—but now that we are in the new world, these tools do make it even more useful. “Codex, write me a script in Guile Scheme to update my home value based on whatever Zillow says it is and record the incremental value as home equity gains” just works like magic. Try getting a web app to do that.]

The initial idea was to make a tool that fit in between two kinds of accounting tools.

1. ledger. A tool that produces reports from specially-formatted text files. It’s very UNIX-y. I like ledger. It is highly extensible because you just need a script that modifies a text file. Easy. I actually first toyed with the idea of making leas an interactive tool to modify ledger files, but I didn’t like the result, and I didn’t find ledger very convenient to use in its natural state. I wanted something more interactive that was still extensible.

2. GUI-tools like GNUCash. I used GNUCash in grad school, but it was a separate GUI app that I’d always forget to open. I wanted something quick and easy from the terminal. GNUCash had some extensibility (in Scheme, too) but not super easy to use.

I also just thought the idea of an Emacs-like personal finance tool was neat-o.

So, I came up with this little structure:

• Interactive C-based command line app.

• The prompt was just a Scheme interpreter that called various Scheme/C functions exposed to Scheme.

My first critical design decision: I realized right away that I did not want to type parentheses all the time:

(command arg1 arg2)

So, I made the interpreter insert the outer parenthesis automatically so that this Just Works (tm):

command arg1 arg2

But, I wanted the program to be more… interactive. I didn’t want to have to type full commands. So, I settled on a UX like this:

> spend
5: Groceries
6: Rent
...
To Account: 5
1: Checking
...
From Account: 1
Amount: 109.23
Description: Walgreens
Year: 2026
Month: 6
Day: 22
>

Of course, I could just code this interaction in C, but I wanted to make the interactive system sort of “universal” and something you could use in Scheme extensions to add new interactive commands. So I came up with this system for making any Scheme function interactive, and also added some special data types like this:

(define add-record
  (lambda ()
    (leas/call "leas/at"
               (list
                (cons "Account" "current_account")
                (cons "Amount" "real")
                (cons "Description" "string")
                (cons "Day" "day")))))

So you can use leas/call to provide arguments to any non-interactive function using data types like “real” and “string” but also with some leas specific datatypes like “day” (for a date), “current_account” for the currently selected account, “account” for an account, “expense_account” (for an Expense account), “liability_account” (for a Liability account), and so on. So now we have some actual accounting data types.

This little structure let me write a small set of core routines and data types in C and then most of the interactive structure in Scheme (the UI, more or less) which was nice.

The basic structure was:

• Book

  • Account

    • Transaction

Each Account has a Type. One of:

1. Asset

2. Liability

3. Income

4. Expense

Buying a McDouble, taking out loans, paying them back, and earning money are all just transfers between accounts. You transfer funds from the Income account to the Asset account on payday. You transfer from a Liability to an Expense account when you pay with a credit card or from a Liability to an Asset account when you take out a loan.

There is this concept of the “current” account, a pointer that attaches to a certain account and marks it “active”. That makes it possible to use commands which just take the current account as their argument which makes it quicker to navigate around. For example, lt lists the last few transactions in the current account.

I use leas every day, so the thing has ended up having lots of niceties. There are commands for doing things like paying off a loan (two transfers: one from an asset account to an expense account to pay interest and one to the liability account to pay off the principal).

I’ve written Scheme scripts to automatically update the values of any funds/stocks/etc. I hold by scraping the internet.

I should point out that this whole “budgeting” thing actually worked. Go figure. Keeping track of spending did let me kind of thread the needle on the wedding. It reduced stress, increased confidence, and gave my skin a healthy glow.

The whole thing is only about 2000 lines of code. Simple, but I’ve used it daily for a long time now. I don’t commit much to the repo anymore. The software is finished, more or less. I fixed an edge case in it a couple of weeks ago. I’m averaging a commit every two years since 2020. That’s the nice part about non-SaaS software. You can finish it. Nothing changes in the environment, and it’s not like any bug is a big deal. It’s running on your machine. You can’t steal someone else’s info if I make an error in the controller.

Of course, there was really no point to this project. The number of budgeting and personal finance tools out there is endless. They are better documented, tested, and so on. I could’ve found something that worked well enough to not be worth the hours I spent on it.

But I had oodles of fun, and there’s the pride of making something useful.

I don’t know shit about fixing cars, so I have to find something else to do.

There’s this terrible piece of advice folks get. If someone’s already solved the problem, don’t re-solve it. Think about another problem worth solving. It’s bad advice. You can’t solve any problem until you learn enough about it.

The best way to learn how the wheel works is to reinvent it.

Oh, and here’s the Github for this project: leas.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

This post will be about experimentation, but there’s a wind-up.

Today, we’re going to look at a class of experimentation problems that we might call “tuning.” A problem where there is a wide variety of plausible choices for a parameter, and we’re trying to find the best one among nearly infinite plausible choices.

Common parameters to optimize include:

• Price

• Promotional Discounts

• Hyperparameters of ML models (search, recommendations, etc.)

Subscribe now

Suppose you want to maximize a function f(u,t). At any point in time, you know f(u_t, t), but you can’t evaluate f at any other points but whatever u happens to be “active”. For example, you know your current prices (u_t) and you know revenue today, but you don’t know what revenue would be at other prices.

How do we learn the best u if we can only evaluate one u at a time?

Let’s think about it as an optimization problem:

Suppose f is strictly concave in u and its gradient satisfies a Lipschitz condition (we’ve decided to be formal-ish today—suit and tie). Then, we could solve this problem via gradient descent, i.e.:

For some suitable sequence of (a).

The problem is: we don’t know the derivative of f, and we don’t know how to evaluate f at other points.

The trick is that we can use experiments to iterate on the gradient descent problem. An experiment is just like computing the objective function and its gradient. It takes a few weeks to compute, though, and our data limits how many points we can compute at once.

The starting point is the relationship between treatment effects and derivatives in this context. Suppose u is a number (not a vector), and you expose half the population to u and the other half to u+h. Well, viola, you have an estimate of the derivative:

[You can expand this to multi-dimensional u by adding more arms to the experiment or by adding structure to f that lets you learn other derivatives in a neighborhood of the current values of u you’re testing.]

First, suppose f doesn’t depend on time, and also, for the sake of simplicity, u is not a vector. Then, the optimal u doesn’t change. It’s static. A stable target you’re trying to hit. Using the gradient descent analogy, we can try to find the best u like so:

When the treatment effect from increasing u is positive, increase u. When it is negative, decrease u, and so on.

There is a vast literature on step sizes that work for gradient descent with a strictly concave f. The key property we need is a method that doesn’t require us to evaluate the function at many points because experimentation limits the number of points we can evaluate. A straightforward approach to choose a, assuming f is strictly concave, using a modern “two-point” method from the literature that works for general concave functions, c/o Malitzky and Mischenko:

Where Df_t is the current derivative estimate, and Df_t-1 is the prior derivative estimate. I’ll call this the MM-rule.

[The MM-rule formula treats dividing by 0 as infinity, so you’ll always choose the other branch of the min{*,*}.]

The main advantage of making this choice instead of the classical gradient descent step size is that you don’t need to run experiments with a ton of arms to approximate the maximization problem:

You’d operationalize this like so:

1. Initially: Run an experiment with the status quo “champion” u vs a “challenger” u + h. Choose h large enough that you’ll see some real action statistically, but small enough that you’re reasonably approximating a derivative. Choose an arbitrary a to get a new challenger. The right choice here is a bit of an art, but it’s fine to just choose it so the treatment looks “reasonable.” This is just an initial condition.

2. Do this again to have enough initial conditions to start using the MM-rule.

3. Now, run the best arm of the previous experiment versus the new challenger generated by the MM-rule.

4. Keep iterating, using the MM-rule above. Repeat step 3 forever.

The nice part about this method of experimentation is that it allows you to make careful, small but continuous iterations toward the truth instead of making large, costly attempts to jump to the optimal value (usually based on strong functional-form assumptions). You’re always testing your latest proposal against the data, so you can’t drift too far.

Practical issues you’ll run into: the formula for a might produce too small of a step size for you to see any real action. The treatments are too close together. It’s fine to put a lower bound on step size, even if it violates some guarantee of convergence. The precision the optimizers look for isn’t attainable if we’re estimating functions from noisy data.

Now, what if the function’s value changes over time, i.e., we have f(u,t)? That’s why we’d run experiments in the first place, instead of choosing different u’s and trying them out to trace out the function. We need to run experiments because otherwise we’ll confuse variation in different u’s for variation in t.

When we experiment, we learn:

So, we can trace out the derivative with respect to u from the data, as in the static function case, but the optimal u we’re chasing is changing as well.

I’m proposing using the same algorithm above to solve the problem when t changes the function (and so, the optimal u)—with a small twist.

Of course, if we’re completely general about how t changes f, this approach won’t work, because the optimal u could jump around wildly, but that’s not really what we think will happen.

Essentially, we’re going to assume that the optimal u varies “slow enough” with t that we can track it by accounting for changes in the gradient with respect to u.

The main practical change is that the MM-rule now depends on function values that we can’t just keep in memory. We need to evaluate the gradients of f(*,t), not the gradients we previously computed of f(*,t-1). So, here’s my simple, practical iteration strategy:

1. Run a four-arm experiment. The experiment has u_t (the current iteration), u_t-1 (the previous iteration), u_t + h, and u_t-1 + h (the h’s can be different if you’d like).

2. Use the results of this experiment to construct L_t.

3. Now, if you look at the formula, you’ll see that technically it depends on a_t all the way back and prior L_t’s that are function evaluations will have been evaluated on f(*,t-2), for example, instead of f(*,t). My sort of pragmatic approach here is to ignore that and use the old a_t’s where they appear as they were when you computed them.

I think this is a reasonable, practical tradeoff to avoid exploding the number of arms the experiment needs while still accounting for the fact that the function has changed, moving from t-1 to t.

Experimentation can test more than button colors. We can use it to understand the models that drive the modern biz. Models that target certain customers with promotional offers or specifically tailored marketing strategies. The number of reasonable models is vast, and finding the right one requires forcing proposed models to confront the truth about their performance. Experiments provide the ground truth and structure the search. It’s a nice pairing.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

All analyses rely on assumptions to map datasets to conclusions. The data doesn’t speak for itself. You have to shape it. Ideally, the mold you fit the data into is based on a reasonable theory of how the world works.

Subscribe now

Analysis is a trick to, with enough data, reduce the world to two potential outcomes:

1. The analysis’s conclusion is true.

2. OR: the assumptions of the analysis are wrong.

This is very useful. It moves the conversation forward. Folks can’t just disagree with your conclusions. They have to disagree with your assumptions, too.

Therefore, an analysis is persuasive only if your assumptions are more plausible than your conclusions.

If your assumptions are more plausible than your conclusions, then disagreeing with the conclusion becomes more difficult after the analysis. Before I could argue against the conclusion directly. Now, I have to argue against the assumptions, which are harder to dispute.

But if your assumptions are less plausible than your conclusions, then you haven’t convinced me. After the analysis, I know either the conclusion is true, or your assumptions are wrong. If I disagreed with the conclusion before, and it is even easier for me to disagree with your assumptions, it is just as easy to disagree with you as it was before your analysis*.

There’s a famous paper in empirical labor economics that illustrates this.

Card and Krueger (1994) claim to show no effect of a minimum wage increase on unemployment.

They do so using a difference-in-differences analysis of bordering counties in Pennsylvania and New Jersey after New Jersey raised its minimum wage. The idea is that these counties are similar to each other and face similar shocks, except that the New Jersey counties received a higher minimum wage. Card and Krueger look at changes in employment and find a small, positive, but insignificant effect. The conclusion is that the minimum wage increase did not cause unemployment. Politicians frequently cite the paper to draw the much broader conclusion that minimum wage increases don’t, more generally, cause unemployment.

It isn’t a persuasive analysis. It’s further confirmation to the believer, but it won’t convert anyone.

Suppose that, prior to the analysis, I disagree with the idea that labor demand is flat or upward-sloping. Now, after the analysis, you tell me: either you have to believe that demand is flat or upward-sloping, or you can believe that these New Jersey and Pennsylvania counties are not really comparable.

Of course, I’m going to conclude that these counties aren’t really comparable. If I have to choose between “People don’t care how much they pay for things” and “These counties aren’t comparable in some subtle way. Maybe another policy or shock that happened at around the same time wasn’t distributed evenly across New Jersey and Pennsylvania counties. Maybe these counties are just wholly incomparable because of a long history of differences in policy and polity between the two states that change how they respond to various shocks. Maybe a silly shock that we’ll never understand happened. Maybe the unemployment effect is delayed because of adjustment frictions.” All of those explanations are more credible and convincing to me than “people don’t care about how much they pay for things.”

So, the study fails to convince anyone. Proponents of a higher minimum wage like the study, but they don’t need to be convinced by it. Sometimes academic economists who find themselves in policy roles where they need to pretend to like minimum wages might find it useful as air cover to explain away past writing that says minimum wages cause unemployment. “Oh, I used to think minimum wages were bad, but then I read Card and Kreuger.” Hot take: No true believer in downward-sloping demand curves has ever been convinced by this. It’s so much easier to conclude that parallel trends don’t really hold or the post period isn’t long enough to see the effect than to invalidate a simple, powerful economic theory on the basis of a diff-in-diff.

This isn’t to argue about an economics paper from more than thirty years ago. It’s to demonstrate that:

Shocking results need mundane assumptions to convert new folks to your way of thinking.

We should be credibility-maxing?

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

*This point isn’t about Bayes’ rule. I understand how my joint prior over the assumptions and conclusions might cause my posterior beliefs about the conclusion to change, even if I view the assumption as less credible than the conclusion in terms of my prior’s marginal distributions. It’s an argument about how people are actually convinced, i.e., I don’t have a tightly constructed joint prior distribution over the likelihood of New Jersey and Pennsylvania counties being comparable and the slope of the labor demand curve. I weigh how plausible the assumptions are and how plausible I think the conclusion is, and that determines whether I end up convinced by the analysis to change my mind.

Ever since I decided to blog well, I’ve been trying to figure out the right way to write a good “links” article. All the cool kids on Substack do it, and I read the Internet. I told myself I ought to be able to do it too. Two statistics blogs and one links blog a week sounded plausible.

Like most things, though, links blogs looked so easy to get right until I tried to do it. So, we’re experimenting with formats. I think I like this one. Two Good Things. Two stories or clips that I think our worth consuming this week. We’ll get there eventually.

This one has bathhouses and AI skeptics.

Subscribe now

Bathhouses

Joe Hovde has a great blog on this company called Bathhouse. It’s a public sauna trying to become a third place—presumably, for the kinds of people who feel comfortable less-than-clothed around strangers: hot people.

A little trend. I’ve noticed that proposed Third Places end up getting accused of being cults—which is natural. They’re trying to replace the Church or the Bar. A Third Place has to have aspects of religion or addiction to satisfy the human need for connection via transcendence or dependency.

I don’t see it crossing generations, but fingers-crossed for them. That’s the key to a successful Third Place. There’s always the old guy at the Bar and the kids with fake ID’s. Church has Sunday school and funerals. Even coffee shops—when those were a bigger deal—cross generations. At least the marketing makes it look like a young single thing?

The website has an “AI Search” feature.

In any case, the article’s a great dive into the business and a category I didn’t know existed before. Check it out.

[I dig this blog in general, by the way. It’s always got some interesting dive into an industry I either don’t know much about or don’t know enough about. Good place to learn stuff.]

“[LLM Companies] Shouldn’t Be Allowed to IPO”

I thought this segment on Bloomberg with Ed Zitron where he hates on AI and AI companies was entertaining, if not persuasive.

Mr. Zitron doesn’t like AI or AI companies, but he has much stronger arguments for the Company’s IPO prospects than the Product and he kind of mixes the two willy-nilly.

The part he’s probably right about is that no one has a great business model yet for how to make money by supplying LLMs (or maybe the market structure just isn’t right yet to do it: you can’t get enough of a margin), and the set of tasks LLMs are good at is much smaller than the AI-positive discourse admits.

Folks are allowed/encouraged/threatened to use AI, and, for many tasks, the price of the tokens is still way more than the value delivered by the LLM. That’s how you get places that had never used AI until a couple years ago, burning millions of dollars to let folks spin up questionable (but expensive) dashboards without knowing SQL or save some Engineer a few (very expensive) hours.

The part he’s (super?) wrong about is this whole idea he often puts out there that the LLM product itself is bad-to-useless. There are nuanced takes on how LLMs can reduce productivity (by yours truly) because workers will substitute towards the LLMs and away from effort even for tasks where effort is more productive than LLMs, but there are other tasks where LLMs absolutely deliver strong productivity gains.

I do not want to go back to debugging without LLMs.

I spent the last weekend debugging this side project (a Rails app), and it would have taken much longer and been so much worse if it had just been me searching for bugs, writing tests, etc, instead of using Codex. I think I spent around 20 hours debugging the app with Codex (i.e. not just prompting “fix all bugs, make no mistakes” like you hear from some portions of the internet…). It easily would have taken me a few weeks without it, and it probably wouldn’t have been done to the same quality because I’m human, and I miss stuff (edge-cases, various long strings of scenarios that cause odd state X, etc).

The LLMs are a little like chess computers when it comes to debugging. Humans miss possible moves in chess all the time—even grandmasters miss mates-in-one in short time control games. Finding all possible one-shot moves on an 8x8 grid is a trivial problem for a computer, though. Similarly, LLMs can come up with ways in which someone could get something funky by hitting a certain string of endpoints much better than my feeble, meatsack mind.

That being said, it is definitely true, and this is Ed’s main point: the value of debugging code, formatting docs, etcetera does not justify the stratospheric level of capital and debt being loaded into these AI Labs. They kind of have to create God or at least replace Raytheon.

I might like (very much) being able to type “replace all my poorly-formatted math in this doc with well-formatted math,” but I’m not willing to pay much for it. I doubt it drives significant company revenue. Or, for example, with the above experience, debugging a web app:

I did it all in Codex (Free).

Yeah, so it was free for me. It wasn’t close to free for OpenAI, I’m sure. How much would I have paid? Not much. It’s just a little side project. If it was a major potential revenue stream? I guess I’d pay a thou or so for access to debugging for a month or something? But I doubt I’d have to. Competition is still too intense. I’d just substitute to Gemini or Claude. So prices stay low…

Until the shakeout stage of the industry (start here for industry lifecycle literature).

I think two things are true:

1. LLMs are super valuable.

2. LLMs are over-valued.

But they won’t both be true when the market for Anthropic’s future dividends becomes liquid and well-informed…

I hope the S-1 becomes public in time for beach reading.

Yes, we do have beaches. Lake Michigan counts.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

A/B tests don’t have to be Experiments. You can use them for other purposes. Corporate political propaganda, for example. Like when politicians cite random studies they haven’t read on a “debate” stage, the point isn’t the study itself. It’s the conclusion.

No judgment. Politics has its place.

The difference between an Experiment and Propaganda is whether you want to learn something or convince someone.

Subscribe now

If you want to convince someone, you will choose metrics that are all but guaranteed to go in a certain direction.

For example, you might remove a screen from a checkout flow and make your metric “time spent in checkout.” Of course, the time spent will go down. You already knew that before running the experiment. You’re building up a case for your preferred product direction, the one that hits your OKR. Not running an Experiment.

If you were running an Experiment, you’d want to know: what do we lose by dropping that screen? What do we gain by dropping it? What metric can I develop that lets both the positives and negatives of dropping the screen shine through, their relative light depending only on how customers behave? Maybe the extra information from the extra screen is worth nothing. Maybe it is everything. The choice of metric should not assume either. It should allow for both cases and let customer behavior decide.

In other words: Experiment metrics maximize surprise. Propaganda metrics minimize it.

Suppose you want to learn something. How do you go about finding a metric?

In the screen-drop example… Suppose the extra screen asks the customer for a bit more information about why they purchased the product. Maybe that information feeds later personalization or targeted messages to get the customer to buy again or subscribe or otherwise take the next step in their journey. It’ll always be something like this. It’s not like the extra-screen was just thrown in there willy-nilly. The folks who added it did so for a reason. We should know why before we knock it down.

We don’t want to use the probability of making it to the end of checkout because, of course, taking away extra screens helps with that.

We also don’t want the metric to be a conversion rate of users to taking the next step. Of course that’s going to fall. The extra screen makes the ad-targeting better.

What we want is a metric that could go either way.

For example, the joint probability of both completing the checkout and taking the next step—not the conditional probability.

If the conditional probability of taking the next step falls, but the number of customers who take the next step increases, then dropping the screen makes sense. You’re improving the probability of checking out without dropping the probability that a top-of-funnel visitor gets to the final stage of the journeys, even though targeting gets worse.

If these two steps have different economic models associated with them (maybe the checkout is the Basic subscription and the extra-screen provides information used to market Premium to the user), then make the metric take that into account. Give each one a dollar value and try to maximize expected dollars.

The point is to get the tradeoff in there. We’re making a move that increases the pr(checkout) and decreases the pr(premium | checkout). So we want to know whether pr(checkout, premium) = pr(checkout) x pr(premium|checkout) increases. We’re surprised either way because, while we know pr(checkout) will increase and pr(premium | checkout) will decrease, we don’t know the net effect.

So, we’ve set up our experiment metrics to learn something, and, along the way, we’ve figured out exactly what we’re uncertain about on the product-side. That’s the right way to select metrics. Like most problems, it can be solved by the algorithm:

1. Write down we know.

2. Write down what we don’t know.

3. Experiment to bridge the gap.

When I was but a wee lad learning to shoot, my dad told me the trick was to “Let it surprise you.” If we anticipate what’ll happen when we pull the trigger, we won’t hit the target.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

Production functions matter. They describe how firms or people turn raw inputs into outputs, i.e., technology. They constrain what’s possible, like demand determines what’s preferable.

Subscribe now

Most empirical work on production functions in economics looks at manufacturing, agriculture, or resource production because their inputs and outputs are easier to measure. But knowledge work is the first job class to be altered by the LLMs. Codex isn’t building bridges anytime soon. So, modeling how knowledge workers produce output and how LLMs affect production matters, but I don’t have any data to work with...

We can learn a lot from theory, though…

LLMs take context (human knowledge) and use it to generate output effortlessly. Without LLMs, humans would still provide their knowledge input, but they also need to supply elbow grease: effort. This blog will show that uncertainty about the best way to do things—and the option to apply effort to counteract that uncertainty—drives how LLMs influence technology choice, production, and the returns to more accurate beliefs.

The plan is for this to be a multi-part blog series with different models of how LLMs transform human context into output.

This post treats LLMs as pure automation machines. They take instructions from a human and act just like a human would, but they save the human significant effort.

The key part of the theory is the role of “uncertainty” in how best to do tasks. It turns out to be critical for understanding the role of LLMs in knowledge production work. We’ll show the following about how uncertainty changes the role of LLMs:

• For low-uncertainty tasks, workers will use LLMs, which will reduce the time it takes to complete a task or improve the quality of their output.

• For moderate-uncertainty tasks, workers will use LLMs, which will increase the time it takes to complete a task or reduce the quality of their output. The worker’s preferred technology will not produce the best output.

• For high uncertainty tasks, workers will NOT use LLMs. Instead, they will use more effort to counteract their uncertainty.

Production without LLMs

Suppose a task can be completed in a certain amount of time, determined (without LLMs) by how smartly people decide to do the task, and the amount of effort they put in to get it done.

[You can think of “time” more abstractly as the inverse of “quality”, where lower time is equivalent to higher quality output, etc. But I’m going to stick to the strict time metaphor for ease of following the model.]

s is the best way to do a task, but it is unknown. Knowledge workers have beliefs (knowledge) about how best to do things, given by:

Without LLMs, the time it takes a knowledge worker to do a given task is given by:

Where z is the way the knowledge worker decides to do things, and e is the amount of effort they put in.

Knowledge workers care about doing a good job, but they also don’t like to put in effort. They choose their effort and the way they want to do the task to solve this problem:

The expectation is with respect to the knowledge worker’s subjective knowledge of the best way to do things. Because the expectation minimizes the square loss, the best z will always be m. By definition, E[(s-m)²] = v. Because this problem is convex for positive e, we can solve for the effort level from the first order conditions:

So, at a high level, uncertainty defines effort. The more uncertain we are, the more effort humans supply to guard against unexpectedly large errors in the course of action they decide to take.

Therefore, the (objective) time it takes a given worker to do a task without an LLM is:

LLMs as pure automation machines

LLMs work by receiving context from humans. Humans can transmit their beliefs to LLMs as context, and then the LLM decides what to do with it, i.e., the LLM production technology is:

Where a gives the equivalent effort level of the LLM’s output. It measures the LLM’s accuracy or performance and might be captured directly by how long it takes to coach the LLM on exactly what you want it to do or on the quality of the output it produces.

Our first model of LLM production is what I’ll call “the pure automation model.” In this model, the LLM performs the task in the same way a knowledge worker would, but effortlessly. It doesn’t have its own beliefs. You tell it what you want done, and it does it. So: f(m,v) = m, because you’ll choose to tell it to minimize the expected time given your beliefs, and the (objective) time it takes to complete the task with an LLM is:

It’s a simple model, but it actually has some semi-non-obvious predictions.

Workers choose to use LLMs when they are MORE certain of the best way to do things

Workers choose to use LLMs-as-automation when they are more certain about how to do things.

The result follows from effort substituting for uncertainty. Workers can put in more effort to mitigate their uncertainty when using the non-LLM technology, but the LLM technology doesn’t respond to their effort.

So, workers choose to use LLMs-as-automation whenever their uncertainty about the best way to do things is low enough:

So, with the introduction of LLMs, only the more ambiguous tasks will be done by the effort technology, and, as the effectiveness of LLMs (a) increases, more and more ambiguous tasks will be done via LLM.

The time to complete tasks increases for some workers when they have the LLM option

This result is “obvious” when you write down a model like this, but not clear at all without actually writing down the math. The key idea is simple, though:

Workers want to avoid spending effort, so they will choose to use LLMs even if the effort technology is faster than the LLM technology.

We already have the rule for when workers will choose to use LLMs. So, to characterize when LLMs increase the time a task takes, we need to look at the conditions under which LLMs are slower:

So, for v = 2a³, the LLM method of production is slower, but workers choose to use it (because 2a³ < 8a³); otherwise, they have to pay the effort cost. Folks don’t just try to minimize time (equivalently, maximize output quality).

For a moderate level of uncertainty, i.e., for a³ < v < 8a³, knowledge workers choose to use LLMs despite the longer task times (equivalently, lower-quality output) they entail.

This production model is static, but it implies something about dynamics. If AI quality (a) improves only slowly, some industries stall in this middle ground, where workers choose to use the LLM technology, but it gives worse/less output. Firms using the effort technology outcompete them. So, AI adoption is quick in industries defined by moderate ambiguity about how best to do things for firms that make it readily available to their workforce. AI adopters lose out to the Reactionaries: firms that force workers to use the effort technology and refuse to pay Anthropic or OpenAI.

Automation LLMs decrease the returns to accuracy if uncertainty is low

Do more skilled workers do better or worse with the option to use LLMs? Do LLMs compress the productivity distribution, allowing less-skilled (maybe early-career) workers to catch up to more skilled workers—or do they widen the gap between the skilled and the less-skilled?

Suppose everyone has the same level of uncertainty (the same v). Then we can define more skilled workers as those with a smaller d = (m - s)².

Shrinking d decreases time by v^(-1/3) for the effort technology and by a^-1 for the LLM technology. So the reduction in time from more accurate prediction is greater for LLMs if:

This is the same as the rule for when LLMs are slower. So, for highly uncertain tasks, the return to skill is greater for the LLM technology than for the effort technology. For low-uncertainty tasks, the return to skill is greater for the effort technology.

If we incorporate the choice constraint into this, we can partition the results like so:

Without LLMs, the return to skill would always be v^-1/3. LLMs increase the return to skill for moderate uncertainty but do not affect it for high-uncertainty tasks (because workers choose effort over LLMs). For low-uncertainty tasks, LLMs are faster than the effort technology and reduce the return to skill.

So, moderate-ambiguity tasks will demand more skilled workers with LLM than they did with the effort technology. Less ambiguous tasks will demand less skilled workers.

So…

If we can compensate for our uncertainty about how best to do something by increasing effort, then we will only do tasks with low uncertainty via the LLM technology. Skill will matter less for low uncertainty tasks. And tasks with moderate uncertainty will now take longer (or be lower quality) than before (tell me you haven’t felt this at some point at work since LLMs came into their own) while demanding higher-skilled workers because of the increased return to skill.

In this model of LLM production, the knowledge worker provides a vital input: their own beliefs about how best to do things. LLMs don’t work without the worker. The model helps us isolate the effects of the automation component of LLMs, but, of course, LLMs do more than automate production (although for some applications… that is pretty close to their purpose).

In the next model of LLM production, we’ll allow the LLM to have beliefs of its own about how best to do things and open up the decision of whether to use a worker at all…

Math…

Although this model of production is relatively simple, it makes arguments that get bandied about in the AI discourse precise and concrete, with at least a plausible theory for what gets done by AI and what doesn’t. Writing down exactly what you are assuming about the structure of folks’ decision-making process clarifies. Implications naturally follow from assumptions and structure.

You can do more than just say things.

Saying, “I think LLMs are great for generating quick queries or searching customer calls, but they’ve made docs less useful because they’re all AI-generated and very hard to parse,” is all well and good, but it’s just like your opinion, man. With a clear model, it starts to make sense.

Writing a good doc is a great example of a moderately uncertain task. It’s not untrod ground, but neither is it exactly obvious how to propose a new idea effectively in a way that accurately communicates what you want to do, why it’s a good idea, and what might go wrong. So, the degradation in quality with the LLM technology available makes sense. It follows from folks avoiding the effort cost to work on a moderately ambiguous problem.

On the other hand, a quick query or labeling customer calls is something you know exactly how to do. It would just take you a relatively long time to do it yourself. And the model says that low-uncertainty scenarios will be better with AI because you can get a baseline effort level without workers having to pay the effort cost for the exact kind of task where the returns to effort (from the worker’s perspective) are low.

The advantage of writing things down precisely in math language is that it makes it clear what assumptions drive your thinking, and it creates natural implications of those assumptions that you’ve not thought of before.

Writing, in both Human Language and Mathematics, is how humans think. Computers have RAM. We’ve got Emacs, Google Docs, 3x5 notebooks, and cave walls.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

We’ve got a few stories from the SaaS wars—and takes, of course. We have an infinite supply of takes, far exceeding demand.

Subscribe now

• Dispatch from the SaaS data access wars:
  Every week, there’s a new story about so-and-so blocking data access from such-and-such. This week, Microsoft is supposedly blocking a certain kind of access from external tools to Power BI. The article contains this haunting quote: “The friction centers on Power BI, a Microsoft product nearly all Fortune 500 firms use to analyze data about their operations in charts and other visual formats.” Really? Nearly all?
  
  I’d like to take this chance to share a conspiracy theory. Power BI isn’t real. I hear about it all the time. It’s allegedly everywhere. And yet… I’ve never used it or seen it be used. I’m tech-old. I’ve been around lots of data environments. Sounds fake to me.

• Did you hear SpaceX is going to be IPO?
  No? I didn’t either. I couldn’t find a single article about it to share. See if you can find one somewhere on the internet.

• Customers demand shorter contracts over uncertainty of “what comes next”
  There’s a story on AI dollars crowding out traditional business software purchases. The pool of dollars isn’t fixed, and if there is a large return to these AI products, why couldn’t these firms borrow to cover token purchases and repay the loans with excess profits? So, something is wrong here. Either credit markets (unlikely), the returns from AI spend are currently negative (plausible until companies learn the right way to deploy it at their firm), or there’s a principal-agent problem (highly likely). The person doing the buying has a fixed budget, so even though the company could just expand borrowing, there’s no cheap way for the agent to pass the information up the chain.
  
  The second half of the story is more plausible, but speculative. Companies are worried that a wave of new, better products will emerge over the next few years and are reluctant to be locked into a long-term contract. That makes sense, but I really haven’t seen any evidence of useful AI-native business software, aside from the models and coding stuff coming from the primary labs. The AI components added to existing SaaS have been useful, but AI features are not really useful on their own. You need something that works well for all the more mundane features of the app first, and that turns out to be a hard problem.
  
  It reminds me of every time I’ve seen what I think is an opportunity to exploit in an industry. I realize that, while I might be right about the opportunity, to actually profit from it, you’d have to launch a competitor that does all the other aspects of the industry right and then just adds the new insight. That’s very hard to do.

• Intuit moves to consumption-pricing.
  Everyone is doing it because it’s the only thing you can do, now that the Variable Cost Revolution is well underway. What is the Variable Cost Revolution, you ask? For a long time, one reason software companies were so profitable was that their costs were almost entirely fixed. Costs did not increase very much as they added more customers. So, with scale, they made oodles of money. But AI has changed that. If you run an AI feature in your software, you have a variable cost product. As demand for the product increases, your costs increase meaningfully. LLM-based features have changed the cost structure of the entire industry, and the pricing structure follows the cost structure.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve been thinking about the fact that statistical modeling doesn’t really have a standard declarative language for specifying a model—and how there’s nothing really preventing that from being so. I have this hope that were such a language to exist, people would finally understand the difference between an estimator and an estimand.

Subscribe now

I started thinking about this recently when I wrote a little something for a side project—coming soon—in STAN.

[I style the name in all-caps for unknown reasons. It’s not an acronym (I think).]

If you’ve never used it before, STAN is a domain-specific language for likelihood models, primarily for Bayesian inference. The core idea is that you can specify the model in this declarative syntax, and the STAN program computes derivatives and the path for Markov Chain Monte Carlo iterations of the posterior distribution using automatic differentiation in a fairly well-optimized way (so, it’s nearly as fast as this sort of computationally-intensive thing can be). It’s super nifty.

To see what it looks like, here is an example of a STAN specification from a model I wrote for an old project.

[If you’re curious: the “old project” is an app that creates an “optimal” schedule for working towards a long-term goal based on your current, observed work behavior (i.e., you like to work on Tuesdays, you like to work in big chunks, you like to frontload or backload work relative to deadlines, etc). You don’t enter your preferences. The app learns from your actual work behavior to help converge on an “ideal” work schedule across projects over time. It never really got any traction, but I liked the idea.]

data {
  int<lower=0> N; // number of days so far
  int<lower=N+1> T; // when the work has to be done
  array[N] real logit_remaining_share; // logit{W / (Y-work prior to W)}
  array[T] int G;
}

parameters {
  vector[6] dow;
  real<lower=0.0> sd;
  real weekday;
  real<lower=0.0,upper=1.0> r;
  real<lower=-0.20,upper=0.20> b;
}

transformed parameters {
  vector[7] K;
  for (i in 1:7) {
    if (i <= 5) {
      K[i] = exp(dow[i] + weekday);
    }
    else {
      if (i==7) {
        K[i] = 1;
      }
      else {
        K[i] = exp(dow[i]);
      }
    }
  }
}

model {
  vector[N] future;
  vector[N] present;
  for (i in 1:6) {
    dow[i] ~ normal(0, 1);
  }
  weekday ~ normal(0, 1);
  r ~ beta(1,1);
  sd ~ gamma(1,1);
  b ~ uniform(-0.20,0.20);

  for (i in 1:N) {
    future[i] = 0.0;
    for (j in (i+1):T) {
      future[i] = future[i] + ((1+sqrt(j-i))^b)*(K[G[j]]^(1/r));
    }
    present[i] = K[G[i]]^(1/r);
  }

  for (i in 1:N) {
    logit_remaining_share[i] ~ normal(log(present[i])-log(future[i]), sd);
  }
}

The key idea here is that you can just describe the prior distributions [weekday ~ normal(0,1)] and the likelihoods [logit_remaining_share[i] ~ normal(log(present[i]) - log(future[i]), sd)] and then STAN simulates the posterior distribution using a fancy Monte Carlo. This makes it easy to specify flexible, realistic priors and likelihoods without relying on computational tricks (like conjugate priors). So you don’t specify models based on computational concerns, but on how well they describe the data-generating process. Plus—and I like this part a lot—the model’s description is fully separate from the details of how it’s estimated.

[Yes, I frequently write code without descriptive variable names. I think this is the good and proper way to write math code when you have a model you’ve written out somewhere in math, i.e., the names here match the LaTex, which is the source of truth, and the real documentation. But your mileage—and ability to get such things past engineers reviewing your PR’s—may vary.]

But I don’t do many Bayesian projects. I mostly work firmly in frequentist land. I don’t do much likelihood-based estimation either. Broadly, I use moment-based estimators, but I really like the idea of a domain-specific language for modeling that’s portable across R, Python, Fortran, Julia, whatever.

I’ve been sketching out what this would look like (I think the cool kids call this “building in public”).

Here’s my current draft:

data
  N;
  K;
  J;
  X[N,K];
  Y[N];
  Z[N,J];
  T[N];
end

parameters
  b[K];
  te[1];
end

latent
  # parameters can show up here
  Y{T} := start |T, t|
       Y if T = t else unobserved;
  end
end

assumptions
  # no parameters or latent variables can appear on the right side of the := operator
  moment(Z @ (Y - X @ b)) := 0; # iv assumption to get b

  moment(te - (Y{1} - Y{0})) := 0; # define treatment effect
  moment(Y{T}) - moment(Y | T) := 0; # exogenous treatment assignment
  
  # alternatively, define some CIA-like assumption
  moment(Y{T} | X) := moment(Y | T, X)

end

Another example for a discrete choice demand model:

data
  J; # number of products
  K; # number of characteristics
  N; # number of customers
  X[J,K]; # product features
  B[N,J]; # dummies for whether each customer bought
end

parameters
  b[K]; # characteristic parameters
  a[J]; # product fixed effect
  choice_prob[N,J]; # customer probability of choosing each option
end

assumptions
  do i from 1 to N
    do j from 1 to J
      choice_prob[i,j] - 1/(1+exp(-1*cross(X[j,:], b) - a[j])) := 0;
    end
  end
  do j from 1 to J
    moment(choice_prob[:,j] - B[:,j]) := 0
  end
  moment((choice_prob - B) @ X) := 0
end

A key point here is that “moment(…)” is not just a function being applied to an input. It’s part of the language, and the language knows the properties of expectations. So it can, for example, know that:

• E[Y{1} | T = 1, X] := E[Y{1} | X] => E[Y{1}] = E[E[Y{1}|T=1, X]]

You can also centralize moments with “central_moment(…)” and get higher order moments by using “moment(X, 2)” or “central_moment(X,2)” (i.e. the variance).

This isn’t a project I’ve written code for yet. I’m still trying to decide whether this makes sense and is useful enough to spend a decent chunk of time on, but in principle, I think having a declarative syntax for moment-based modeling makes sense.

The core, workhorse estimator behind it all would be the Generalized Method of Moments, because it’s flexible enough to cover all of these cases. Maybe it can even include inequality restrictions…

Anywho, thank you for listening to me ramble about ideas I’ve been bouncing around to improve statistical programming. I really like the idea of more firmly decoupling the estimand from the estimator in the code itself, so I’ll probably take a shot at making this a reality once I’ve thought about the problem more precisely and have a good grasp of how to implement it right.

What do you think are the biggest gaps we have in statistical programming?

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I would have liked prediction markets if I’d thought about them much before I saw them in the wild.

I’m a fan of markets. Why not learn what people really think by having them put money on the line? I do like forcing pundits to bet. We should tax bullshit. It has negative externalities.

But now, we have data, and, unfortunately, we have to use it.

Subscribe now

Another argument I would have found convincing in the past: who cares if there’s insider trading? In fact, that will just make the markets better at predicting because all information, including that available to insiders, gets incorporated into the price.

The idea that information is valuable is central to economics and intuitive. The more information is reflected in prices, the better decisions we can make about capital allocation. The misallocation of capital is a major drag on productivity.

A few things were missing from my model of the world:

1. The value of information is heterogeneous.

2. The value of information does not correlate with market size.

3. Not all information is disperse and diverse, which is where markets shine.

4. There are, naturally, negative externalities to gambling. So, the market's social value does not start at 0.

For example, one of the larger prediction markets was a bet on who would win the New York City mayoral race. What is the value of this information before the election concludes? Perhaps it provides a means to hedge against a certain candidate’s effect on asset prices if he wins? Or, aggregate information from this market’s outcome can be used to trade in the asset market.

Markets do not create information. They reveal and aggregate the information people bring to them. So, what additional information is the market aggregating here? It can’t come from public polls, which are, well, already public. It does generate a numerical index of the “vibes,” which are public knowledge but difficult to quantify. It’s not clear how predictive the vibes are, though, and, in any case, they are already reflected directly in asset prices. From insiders, it incorporates private polling data, which is probably the most useful information generated by the market.

But couldn’t those insiders use that private polling data to directly trade on assets that will (allegedly) be affected by a certain candidate’s win? Then, that information would already be in asset prices.

No, is the answer, I think, but the reason for that is that political insiders are small in asset markets, so they can’t really move the price, but large in (some) prediction markets, given their current size.

[I have an armchair theory that media fascination with prediction markets is that polls report their results in terms of proportions, while prediction markets return results that give an implied probability of so-and-so winning. Of course, polls could also return probabilities of winning with some kind of Bayesian inference, but they don’t. An opportunity for a polling company.]

And therein lies the reality of the information value of prediction markets: it depends on people with meaningful information making up a sizeable enough percentage of the market that their private information gets reflected in prices. If the only people with meaningful information are insiders and insiders are small, then they earn profits by trading in the market because their bets don’t move the price, but we don’t learn any special information from the market that we don’t already know.

The prediction markets that would be most useful are not political horse races or sports gambling, but predictions about events where the information is difficult to quantify and aggregate into a belief in the abstract, and where information is diverse and widely distributed.

For example, the market for apples aggregates information on the value of apples. Everyone has their own apple preferences, so the information is very diverse. No one else knows your preferences, so the information is widely distributed. Markets aggregate all this diverse and dispersed information into a price for apples, which summarizes how apples trade off versus any number of other goods.

Markets like these are not popular topics on Kalshi. Look at the home page. I struggled to find topics where many individual people have meaningful, diverse information about them. I’ll discuss a few that were (arguably) close to that before jumping into what’s mostly there.

• “Will Americans receive stimulus checks?” Of course: how would any non-insider have useful information to contribute to the market here? But, it does, I suppose, operate as a hedge of some kind in case the government has to borrow to write stimulus checks, and so it might offer some idea of how exposed people are to the demand for treasuries or something (which is diverse, dispersed information)… but there are already so many ways to do this that have a market size of more than $1.5M.

• And, sure, there are some markets here that are more or less just typical financial instruments. There’s one directly betting on when Bitcoin will cross $100k again, which is just a typical financial market that aggregates individual demand information. But, again, this isn’t and wasn’t really the pitch for prediction markets. This is just a normal financial market.

• “More tech layoffs in 2026 than 2025?” While there’s some alpha from being an insider, it’s limited by the fact that you only know your company, and the market can effectively aggregate all these signals about the tech industry more generally. Workers without direct knowledge of pending layoffs do have some information on them. You can usually tell when it’s coming, and such a market might be a hedge against the possibility you’re laid off. Others outside the tech companies might have some useful leading indicators worth aggregating as well, e.g., that it’s getting harder to close commercial real estate deals in the Bay Area.

The combined market size for these and a few other roughly information aggregative markets is dwarfed, of course, by the $293M market for “PGA Championship Winner,” which is what these sites are really about.

“For where your treasure is, there your heart will be also.”

If you write down a typical model for a prediction market, you’ll start saying things like “each individual gets a signal x…” but they don’t really. Not in most of these. As much as I’d like to believe I have some signal about what’s going to happen with geopolitics, I don’t. My beliefs on the likelihood of various outcomes of the war in Iran are just an index of public information. I have no new signal to offer the market.

So, then, what’s left? If we don’t really draw signals, then we’re left with non-insiders really just placing bets on the events because they are risk-loving, either because they are in a relatively flat part of the wealth utility space and need to get a great outcome to move up a level, or because they have “an innate preference for risk.”

Go to Kalshi and ask yourself: what price in each of these markets would cause me to enter? Most of these, within a very broad, reasonable range, I would have no reason to think are mispriced, even if the price were like +/- 20% from what it is currently. I’d just assume the market had more information than I did.

The same isn’t true in the stock market. For example, if the price of Apple were 20% lower than it actually is, I could look at their financials, public roadmap, and go, “Wait a second. This is wrong. I am going to make a boatload of money. Can I take out a second mortgage on the house?”

So, this is why I think prediction markets have failed in reaching their promise:

1. For most of the predictions we want the markets to make, relevant information is highly concentrated.

2. There are very good substitutes for the more meaningful markets that directly focus on more traditional financial transactions and don’t rely on people with “innate preferences for risk” to get liquidity.

I’m not sure that we should like a system that functions as a transfer payment to insiders from people who think the key to improving their lives is to put it all on black.

I’m not for banning prediction markets, of course. People should generally be free to do as they will. Many vices and antisocial behaviors are legal.

We still let people be Dodgers fans, among other things.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

This was a strong week for pricing news, so I’m leaning into it with a round of a few choice pricing/monetization/data/statistics stories from the week. Maybe it’ll be a recurring thing (actually recurring, not “recurring” as in ARR).

Subscribe now

1. Some Anthropic customers don’t like its pricing. I heard not so long ago that AI was ending scarcity and replacing white collar work by 2030 at the latest. Now, even the Silicon Gods have a price, and some folks think it is too much. Just imagine if the robots cost more than our glorified address book! That’s the contradiction of LLMs. They’re fantastic products—amazing, in fact, but if they were as good as the AI labs’ frontmen say… they’d cost more. They’re more than worth it at current prices, of course. “Many technology firms and other large Anthropic customers say they plan to eat the soaring costs...” Price competition is tough because of relatively low product differentiation between the top players, which should keep prices down… but even Google can’t light money on fire forever...

2. Speaking of the cool, currently cheap-enough AI… I haven’t had the chance to play around with it yet, but the demos of Hex’s new “generative apps” look snazzy. If you don’t know Hex, it’s a notebook ala Jupyter that makes it easy to spin up web apps powered by the notebook for your stakeholders’ consumption. It’s well-done, and the stylings of these new vibes-powered apps look fun.

3. Spotify is running this hilarious monetization play. If you open Spotify in the browser and switch tabs, the miniplayer pops up in the lower-right. If you are not on Spotify Premium, like me, it shows the miniplayer covered with a prompt reading, “You discovered a Premium feature. With Spotify Premium, you can make the miniplayer even smaller.” I had never noticed the size of the miniplayer before. It had never occurred to me that it was too big, but after I clicked “Not Now” and the player ballooned to its previously normal size, I saw how large it truly was. Like Adam in the Garden, I had tasted the fruit of a properly-sized miniplayer, and lost my innocence. I felt a sense of loss of what was once and what could be again… if only I didn’t refuse to pay $120/yr or whatever Spotify Premium is. I don’t mind ads. It’s nice to know what the fellow corporates are up to.
   
   Seriously, a very clever surface to drive conversion on. I love it. I bet it works.

   

4. Slowly but surely, AI app pricing is matching the AI cost model. In the long run, cost structure sets pricing. Take Figma, for instance. They started capping how much customers can chatter with the Clankers for a flat rate, like the old cell phone plans. Now, you buy credits on top of the seat price once you cross a certain usage threshold. This always had to be the case, eventually. It’s not just Figma, of course.

   

5. These rectangles sold for $85.8 million, so next time, Just Monetize It (tm). Someone will buy it.

   

The Information owes me a referral fee for dominating the links, but they had great stories this week. Nice job, folks!

Blog and Side Projects Updates

• Crossed 100 subscribers this week. Super-Neat! Hope you enjoy reading!

• Completed our first full week of the new, thrice-weekly posting schedule! I want to write well, and it turns out you have to practice. Go figure.

• We made a small chunk of progress on the last main project area for my secret side project (a webapp for this Common Statistical Task). We got to write some Stan for the first time in a while.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve complained about the demonization of pricing experiments before, but alas, there’s no arguing religion. Sometimes, you just have to accept that you can’t run price experiments, and still, They want you to estimate demand.

The trick, of course, is to convince Them to let you randomly send some folks a 10% off coupon for whatever you’re hawking. Often, the marketing/promo team has already done this, so you can just steal their data. This is best.

The idea here is that the promo introduces price variation, but you have to be careful with how you use the data to identify demand. So, thought I’d write a little post on how to do it. Price well!

Subscribe now

A simplified empirical demand model (ignoring all the nonsense you control for, etc) looks something like this:

Where j indexes products and t indexes time.

The problem is that historical price changes are related to all the other factors that affect output (E). For example, often this equation more or less describes the price variable:

i.e. the price increased at some point in time in the past. It is extremely unlikely that nothing else changed with demand before and after the price change. Demand is always changing.

So, you can’t just run the regression.

The standard solution to this problem is to use an instrument that is uncorrelated with the other factors that drive output (e.g., demand shocks), but is correlated with price, because then:

So, you can just solve for the elasticity β.

But you don’t have any instruments. You know this. I know this. The supply of instruments is much lower than the demand, and, if you’re working in industry, it’s just extremely unlikely that some natural experiment exists for your particular company.

So, you have to make your own instruments. That’s what experiments do. Experiments are instrument factories. They make random variables that are uncorrelated with everything but what comes after being bucketed into the experiment.

But the 10% off coupon mentioned earlier isn’t a valid instrument for this demand model, sadly.

It’s randomized across users, not products. So, there’s no reason to suspect it’s excluded from a direct shock to demand. The randomization ensures that variant assignment at the individual level is uncorrelated with other user characteristics, but this doesn’t solve the endogeneity problem, which is cross-product and time!

The problem is that this demand model is based on product-time variation. So, to use this instrument, we’ll need to build up a customer-level demand model and derive the aggregate demand for each product from the individual demand curves.

A natural way to do this is with a discrete choice model. To save on notation, let’s suppose you’re hawking one product, so that this amounts to modeling the conversion rate. Here’s a pretty standard model, but you can use something more flexible:

Where X_i are some customer characteristics and P_i is the final price of the product for customer i. Here, we’re allowing for other reasons a particular customer might have a lower price aside from access to the 10% discount. P_i is just whatever the customer’s final price ends up at.

Now, let Z_i = 1 if the customer gets the 10% promo and 0 otherwise. The key idea is to use this Z_i as an instrument for price.

For doing binary choice models with instrument variables, I usually use a little something I wrote circa 2017-2018: repeated two-stage least squares.

The basic idea is that you can show that doing the following procedure is a contraction mapping:

1. Run two-stage least squares for the linear probability model of 1(purchase_i) on (X, P) with (X, Z) as instruments. Call this estimate c.

2. Set k = 1.

3. For some initial guess of a and b, compute the predicted conversion rate. Run the same two-stage least squares estimation with the dependent variable being the predicted conversion rate. Call this estimate c_k.

4. Update: [a_k+1, b_k+1] = [a_k, b_k] + c - c_k.

Under an assumption that the instrument has a “strong enough” relationship to (X,P), this procedure converges to the true a and b in a large enough sample. It’s also got standard normal, asymptotic inference because there’s an analogous GMM estimator.

The key thing is that you can use these kinds of experiments to generate plausible instruments for demand estimation even when you don’t have the ability to do more direct price experimentation (for whatever reason).

Once you have this model, you can simulate changes in price by just plugging in new prices, but identification here has its limits. All you’ve done is exogenously moved prices 10% for some customers. If you start looking at 30% price increases, you’re inferences will be based more on functional form than anything in the data. The real info you want to get is “Are we over/under-priced? Would a small increase/decrease help us?”

A couple things to try to improve power and flexibility here:

1. If you can get more promo depths (lower or higher discount %), that’s super useful because it will give you more instruments. Maybe you can get a more complex model of the pricing curve. With only the one 10% promo, we can’t introduce a P² term into the model because we don’t have a separate instrument for it. But if we have a few different treatments, we can get it in there.

2. If you can get them to send out the promo a few times where you re-randomize the recipients, this is good, because it generates additional variation in variant assignment, giving you more power.

There’s an important wrinkle here. The e-mail or notification that tells a customer they have a promo itself might be responsible for increased demand, i.e. aside from any discount the customer gets.

The trick to deal with this is to send promos to everyone but vary the discount level. Instead of doing 10% vs 0%, do 20% vs 10%. Then, you can send the same e-mail to everyone and just change the discount. This helps isolate just the pricing impact of the experiment.

At this point, we might ask: “Well, if we’ve got this nifty trick to run a more palatable experiment, why would we ever want to run pricing experiments?” This is another case where it’s important to remember that the data’s only really informative in a neighborhood of actual prices people are seeing. This strategy only tests prices below the status quo, not higher. So, you’re not really getting direct evidence on what would happen at higher prices. You’re just assuming that the elasticities you see for the 10% lower price are roughly relevant when the price is 10% higher. That’s better than vibes and purely observational analyses, but it’s an assumption all the same. So, you have to be more cautious when increasing prices than decreasing them using this method.

Thanks for reading!

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve become less skeptical of surveys. Economics, as a field, has a bias for revealed preferences—what people do—over what they say, and I mostly inherited the biases of the field because, by default, you believe whatever you read when you were twenty-three.

But I’m becoming a fan of surveys.

This post is about both sides of the revealed vs. stated-preference divide. I talk about the problems with using surveys to package and price products and ways to replace some common survey use cases with revealed preference methods. But also about why sometimes it pays to just ask people what they’d do, and believe them.

Subscribe now

The difference between saying “I’d pay $100” and writing a check for $100 is the difference between the plan to run a marathon and lacing up. This is the fundamental problem with surveys.

Conjoints don’t solve this problem. Artificially creating a realistic choice set doesn’t bridge the gap between stating and paying.

[My little take here is that for the narrow task of pricing (as opposed to understanding the determinants of demand), conjoints are worse than just asking “would you pay $5?” My evidence is classified, but I’ve seen many price elasticities from conjoints, and they are always wrong to the point of being useless, while I’ve seen decent-enough results from the “would you pay $X for ___?” question.]

The right argument for running a survey is that we don’t have a good answer to the question: “what else could we do?” When there’s nothing else to do, you run a survey. But there’s a temptation to throw in the towel early because (we imagine) a survey is a time-bound, well-defined task that we can probably ship in a week or two.

But you can learn a lot from revealed preferences before you have to resort to stated preferences.

It’s rare to launch a genuinely new product, a product that’s completely unrelated to whatever else your company does, so you don’t have any relevant data. Most new products are re-packaging your existing product or a small new add-on. You want to offer an all-you-can-eat subscription, alongside your current à la carte, transactional business. You’re going to start charging for a type of product usage that was previously free. Etcetera.

When revealed preference data exists that answers the relevant question, it’s better because it represents real checks. For example, it might look like you don’t have any data on how people will respond to a price for a previously free product, but you can triangulate.

• When you introduced this product, how many more people signed up for the subscription?

• When you raised the price by $5 last year, how many fewer people did?

Combining those two answers tells you something about how valuable customers find the feature. You can say things like “people value the feature like a $3 reduction in the product’s price, so…” [You do need both parts here: variation in the presence of the feature and variation in the price.]

The point is that, if you’re a wee bit creative, you can use data to answer a broad class of questions before you need to make your own data via surveys.

Surveys are worse than behavioral data, but they are 1000x better than vibes (the 1000x, however, was measured by vibes).

Analysis is a competitive market. If there’s no data analysis or survey, decisions will be made via vibes. There’s always an outside option. People don’t have to use any evidence at all to make decisions. You can just decide things. Vibes are always a backup plan.

[This is why I lean towards “give stakeholders something” instead of “we don’t really have any ready data on that” or “oh, that’ll be involved, we might be able to work that into the next sprint” (at which point the data is, to a rough approximation, always useless: the decision’s been made). The question is: “Do you have something that’s probably better than vibes?” If so, ship it.]

The point of a survey is to create data where we don’t have it. The typical types of missing data are:

1. We don’t have price variation. The price has never changed before.

2. We don’t have product variation. Either everyone has always had the product, or no one has ever had it. So we don’t know how much anyone will like it.

3. We have no way to connect the new product to an existing product (it’s not just a re-packaging of goods we’re already selling).

It’s surprising how many problems in statistics amount to missing data problems. Causal inference is all about the fact that we don’t observe the potential outcome for treatments that people never experienced. Pricing new products is the same kind of problem. We don’t know what people will pay for products they’ve never been offered.

This is the key frame to use when writing surveys: what data is missing? We don’t just want to run survey to run a survey. We have a goal. We’re trying to fill in the gaps in our data. Answering this question makes writing a focused, short survey easy and natural. We need to ask questions that extract the missing data.

We don’t have price variation? Then we need to ask people, “Would you buy it at price X?” and “Would you buy it at price Y?” Bam. Now the price varies between X and Y.

We don’t have product variation? Suppose we’re looking at adding a new feature to a bundled subscription. Then, we need to ask the questions “Would you buy it at price X if it included feature Z?” and “Would you buy it at price X if it didn’t?” i.e., we need to keep the pricing constant and vary the feature set.

Another kind of missing data we need to extract from a pricing survey: the respondent's size. If you’re selling, say, a B2B product, the size of the company purchasing the product will be relevant to how much this customer’s price sensitivity matters. If you’re doing seat-based pricing, then you need to know the number of seats the respondent represents. If you’re selling a transaction-based or usage-based consumer product, you have to get a sense of how many potential transactions or how much potential usage the respondent represents. The particulars of how to do this depend on how demand scales for the product, but it’ll usually be something like “how many times do you run workflow X” or “how many employees work at your company”.

From there, you can maximize revenue/profit (depending on your cost structure) weighted by the respondent’s demand size (a lot of survey tools don’t do this sadly: they’ll just maximize revenue for you without taking into account demand size—alas). The demand curve is just given to you by the highest prices that people say they’re willing to accept.

I like to solve a problem that looks like this:

Where P is the price, W(i) is the weight associated with the size of the respondent’s demand (maybe based on their answer to company size questions, etc), WTP(i) is the highest price folks say they’re willing to pay (the survey usually gives an upper and lower bound on WTP, so it makes sense to solve this at each of those extremes), a is a discount or premium on people’s stated willingness to pay. I like to check for robustness to deviations between stated willingness to pay and true willingness to pay, or even just to downweight WTP if you’re trying to price for growth. So you can plot the optimal price as a function of a and find a price that doesn’t vary a lot for small deviations in a.

There’s a bit of an art to this.

For a well-targeted survey, a very high percentage of responses to the “Any other thoughts?” prompt are really thoughtful. This isn’t a real quote, but it’s illustrative of responses you’ll get: “Seat-based pricing wouldn’t really track the value here. We’d really only need one seat because only one engineer needs to manage this. The usage option you presented makes more sense, but only if there’s some cap or it gets cheaper as we scale…”

Humans like being helpful, and even though it takes them time, and it doesn’t really affect whether they get the $25 gift card, respondents take surveys seriously. Because humans are generally good folks who like to help their fellow corporate drones out. Go figure. The kids call this a “white pill” (I think?).

It’s not like you have to blindly roll out the survey’s optimal price. A couple of in-depth interviews with potential customers can at least make sure you’re not going off the deep end. If someone says, “That sounds reasonable,” it probably is. If they say, “lol what?” something’s probably gone wrong.

You don’t have to completely discard vibes. Vibes contain useful data. They’re a statistic of past experience. So, you can use them to discipline the survey, while still basing pricing (or other decisions) on data. If some respondents’ stated willingness to pay makes no economic sense, just drop them. If you’re worried people are overstating their willingness to pay—before they have to actually convince Finance to sign off on it—then just underprice it (set a < 1 in the above problem).

The point is that you can build trust in survey results. You don’t have to treat them as a machine that takes responses and converts them to decisions. This makes them infinitely more usable, despite their limitations.

I’ve been thinking a lot about surveys recently. There might be a few more survey-focused techniques in upcoming blogs (thinking through some issues).

Zach

Connect at: https://linkedin.com/in/zlflynn

I’ve decided to write well.

Now, I know the first step traditionally is to move to Paris or a small American town—you can get away with NYC if you’ve got old money—and preferably not be employed in B2B SaaS, but I don’t have that kind of dedication. I like money, I have suburban tastes, and I like my neighbors like I like my comments—anonymous.

So, I’m going to try another tactic: practice.

Subscribe now

I’ve been blogging once or twice a month on statistics (which you call “data science” when you’re trying to get a job) for a few years now, but I’m going to write more frequently now. My goal is to put out a blog three out of every seven days.

Of course, that doesn’t mean I’ll write up three experimentation techniques every week, like in the old blogs. I have a limited supply of these! But I realized I had a lot of thoughts about industry tools, practices, news, etc., and just listing the ideas in my little Kanban board made it clear there was enough content to do it.

So, we’re broadening our horizons, limited by the following principles:

1. I have to care about the topic.

2. I have to know enough about it to not sound stupid (hard!).

Especially looking forward to doing more tools-of-the-trade reviews! I did one recently on ggsql. There’s a ton of products, tools, techniques, etc that I’ve never really used because they weren’t used at prior jobs—or, they’re new—and it’d be fun to take a look at them.

I’ve got some initial ideas, but feel free to pitch me on tools to review! Including ones you wrote! (With the warning that I will say what I think of it. My rate for lying about how great your tool is $1 million. So readers should know that either I think the tool is really good or you paid me $1 million.)

Looking forward to the challenge! My current plan is to write the posts mainly over the weekend and schedule them throughout the week, but we’ll see how it evolves…

Zach

Connect at: https://linkedin.com/in/zlflynn