Effort Post: Automation, Knowledge Work, Uncertainty, and AI (Part 1)
A production model where LLMs act as pure, effort-saving automation machines (more models upcoming!)
Production functions matter. They describe how firms or people turn raw inputs into outputs, i.e., technology. They constrain what’s possible, like demand determines what’s preferable.
Most empirical work on production functions in economics looks at manufacturing, agriculture, or resource production because their inputs and outputs are easier to measure. But knowledge work is the first job class to be altered by the LLMs. Codex isn’t building bridges anytime soon. So, modeling how knowledge workers produce output and how LLMs affect production matters, but I don’t have any data to work with...
We can learn a lot from theory, though…
LLMs take context (human knowledge) and use it to generate output effortlessly. Without LLMs, humans would still provide their knowledge input, but they also need to supply elbow grease: effort. This blog will show that uncertainty about the best way to do things—and the option to apply effort to counteract that uncertainty—drives how LLMs influence technology choice, production, and the returns to more accurate beliefs.
The plan is for this to be a multi-part blog series with different models of how LLMs transform human context into output.
This post treats LLMs as pure automation machines. They take instructions from a human and act just like a human would, but they save the human significant effort.
The key part of the theory is the role of “uncertainty” in how best to do tasks. It turns out to be critical for understanding the role of LLMs in knowledge production work. We’ll show the following about how uncertainty changes the role of LLMs:
For low-uncertainty tasks, workers will use LLMs, which will reduce the time it takes to complete a task or improve the quality of their output.
For moderate-uncertainty tasks, workers will use LLMs, which will increase the time it takes to complete a task or reduce the quality of their output. The worker’s preferred technology will not produce the best output.
For high uncertainty tasks, workers will NOT use LLMs. Instead, they will use more effort to counteract their uncertainty.
Production without LLMs
Suppose a task can be completed in a certain amount of time, determined (without LLMs) by how smartly people decide to do the task, and the amount of effort they put in to get it done.
[You can think of “time” more abstractly as the inverse of “quality”, where lower time is equivalent to higher quality output, etc. But I’m going to stick to the strict time metaphor for ease of following the model.]
s is the best way to do a task, but it is unknown. Knowledge workers have beliefs (knowledge) about how best to do things, given by:
Without LLMs, the time it takes a knowledge worker to do a given task is given by:
Where z is the way the knowledge worker decides to do things, and e is the amount of effort they put in.
Knowledge workers care about doing a good job, but they also don’t like to put in effort. They choose their effort and the way they want to do the task to solve this problem:
The expectation is with respect to the knowledge worker’s subjective knowledge of the best way to do things. Because the expectation minimizes the square loss, the best z will always be m. By definition, E[(s-m)2] = v. Because this problem is convex for positive e, we can solve for the effort level from the first order conditions:
So, at a high level, uncertainty defines effort. The more uncertain we are, the more effort humans supply to guard against unexpectedly large errors in the course of action they decide to take.
Therefore, the (objective) time it takes a given worker to do a task without an LLM is:
LLMs as pure automation machines
LLMs work by receiving context from humans. Humans can transmit their beliefs to LLMs as context, and then the LLM decides what to do with it, i.e., the LLM production technology is:
Where a gives the equivalent effort level of the LLM’s output. It measures the LLM’s accuracy or performance and might be captured directly by how long it takes to coach the LLM on exactly what you want it to do or on the quality of the output it produces.
Our first model of LLM production is what I’ll call “the pure automation model.” In this model, the LLM performs the task in the same way a knowledge worker would, but effortlessly. It doesn’t have its own beliefs. You tell it what you want done, and it does it. So: f(m,v) = m, because you’ll choose to tell it to minimize the expected time given your beliefs, and the (objective) time it takes to complete the task with an LLM is:
It’s a simple model, but it actually has some semi-non-obvious predictions.
Workers choose to use LLMs when they are MORE certain of the best way to do things
Workers choose to use LLMs-as-automation when they are more certain about how to do things.
The result follows from effort substituting for uncertainty. Workers can put in more effort to mitigate their uncertainty when using the non-LLM technology, but the LLM technology doesn’t respond to their effort.
So, workers choose to use LLMs-as-automation whenever their uncertainty about the best way to do things is low enough:
So, with the introduction of LLMs, only the more ambiguous tasks will be done by the effort technology, and, as the effectiveness of LLMs (a) increases, more and more ambiguous tasks will be done via LLM.
The time to complete tasks increases for some workers when they have the LLM option
This result is “obvious” when you write down a model like this, but not clear at all without actually writing down the math. The key idea is simple, though:
Workers want to avoid spending effort, so they will choose to use LLMs even if the effort technology is faster than the LLM technology.
We already have the rule for when workers will choose to use LLMs. So, to characterize when LLMs increase the time a task takes, we need to look at the conditions under which LLMs are slower:
So, for v = 2a3, the LLM method of production is slower, but workers choose to use it (because 2a3 < 8a3); otherwise, they have to pay the effort cost. Folks don’t just try to minimize time (equivalently, maximize output quality).
For a moderate level of uncertainty, i.e., for a3 < v < 8a3, knowledge workers choose to use LLMs despite the longer task times (equivalently, lower-quality output) they entail.
This production model is static, but it implies something about dynamics. If AI quality (a) improves only slowly, some industries stall in this middle ground, where workers choose to use the LLM technology, but it gives worse/less output. Firms using the effort technology outcompete them. So, AI adoption is quick in industries defined by moderate ambiguity about how best to do things for firms that make it readily available to their workforce. AI adopters lose out to the Reactionaries: firms that force workers to use the effort technology and refuse to pay Anthropic or OpenAI.
Automation LLMs decrease the returns to accuracy if uncertainty is low
Do more skilled workers do better or worse with the option to use LLMs? Do LLMs compress the productivity distribution, allowing less-skilled (maybe early-career) workers to catch up to more skilled workers—or do they widen the gap between the skilled and the less-skilled?
Suppose everyone has the same level of uncertainty (the same v). Then we can define more skilled workers as those with a smaller d = (m - s)2.
Shrinking d decreases time by v(-1/3) for the effort technology and by a-1 for the LLM technology. So the reduction in time from more accurate prediction is greater for LLMs if:
This is the same as the rule for when LLMs are slower. So, for highly uncertain tasks, the return to skill is greater for the LLM technology than for the effort technology. For low-uncertainty tasks, the return to skill is greater for the effort technology.
If we incorporate the choice constraint into this, we can partition the results like so:
Without LLMs, the return to skill would always be v-1/3. LLMs increase the return to skill for moderate uncertainty but do not affect it for high-uncertainty tasks (because workers choose effort over LLMs). For low-uncertainty tasks, LLMs are faster than the effort technology and reduce the return to skill.
So, moderate-ambiguity tasks will demand more skilled workers with LLM than they did with the effort technology. Less ambiguous tasks will demand less skilled workers.
So…
If we can compensate for our uncertainty about how best to do something by increasing effort, then we will only do tasks with low uncertainty via the LLM technology. Skill will matter less for low uncertainty tasks. And tasks with moderate uncertainty will now take longer (or be lower quality) than before (tell me you haven’t felt this at some point at work since LLMs came into their own) while demanding higher-skilled workers because of the increased return to skill.
In this model of LLM production, the knowledge worker provides a vital input: their own beliefs about how best to do things. LLMs don’t work without the worker. The model helps us isolate the effects of the automation component of LLMs, but, of course, LLMs do more than automate production (although for some applications… that is pretty close to their purpose).
In the next model of LLM production, we’ll allow the LLM to have beliefs of its own about how best to do things and open up the decision of whether to use a worker at all…
Math…
Although this model of production is relatively simple, it makes arguments that get bandied about in the AI discourse precise and concrete, with at least a plausible theory for what gets done by AI and what doesn’t. Writing down exactly what you are assuming about the structure of folks’ decision-making process clarifies. Implications naturally follow from assumptions and structure.
You can do more than just say things.
Saying, “I think LLMs are great for generating quick queries or searching customer calls, but they’ve made docs less useful because they’re all AI-generated and very hard to parse,” is all well and good, but it’s just like your opinion, man. With a clear model, it starts to make sense.
Writing a good doc is a great example of a moderately uncertain task. It’s not untrod ground, but neither is it exactly obvious how to propose a new idea effectively in a way that accurately communicates what you want to do, why it’s a good idea, and what might go wrong. So, the degradation in quality with the LLM technology available makes sense. It follows from folks avoiding the effort cost to work on a moderately ambiguous problem.
On the other hand, a quick query or labeling customer calls is something you know exactly how to do. It would just take you a relatively long time to do it yourself. And the model says that low-uncertainty scenarios will be better with AI because you can get a baseline effort level without workers having to pay the effort cost for the exact kind of task where the returns to effort (from the worker’s perspective) are low.
The advantage of writing things down precisely in math language is that it makes it clear what assumptions drive your thinking, and it creates natural implications of those assumptions that you’ve not thought of before.
Writing, in both Human Language and Mathematics, is how humans think. Computers have RAM. We’ve got Emacs, Google Docs, 3x5 notebooks, and cave walls.
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn