Memo-Generating Machines
Why memos are better than dashboards, and how, at last, they will replace them
Dashboards are the worst form of self-service analysis, except for all the other forms that have been tried from time to time...
They are terrible because they are inflexible. Dashboards take material time to build, yet they don’t answer 85% of the class of questions they were supposed to answer. The questions are ever-changing, while the dashboards are static, fixed. They need another filter. Another cut. Another unit. Another metric. So, what becomes of the dashboard? Reactively, we expand its filters and parameters. Still, it sits there unused, and folks continue to ping their friendly neighborhood analysis person to do whatever needs doing.
And then: what does the analysis person do? They don’t send back a dashboard-like document, a sentence of text around charts and tables. No. They write a memo [a “doc”], describing what they found and what the analysis means.
The point of dashboards is to enable self-serve analysis, yet they look nothing like the analyses data scientists produce. Think about how weird it would be if a data scientist just sent you back a couple of graphs when you asked them to look into something. There should be a narrative, an argument, facts… Something.
Why should self-serve analysis look different than the analysis I do?
The only difference between self-serve analysis and bespoke, artisanal, handcrafted analysis from the Analyses region of Northern Illinois should be who ran the thing.
Self-serve analysis has a great premise. The business questions come from one set of humans, but the answers traditionally require another set of humans, who (on occasion) have to eat, sleep, and play Cyberpunk 2077. What if we could scale the answers? What if we could write the analysis once and scale it so that we could automagically generate answers to a broad class of questions? If only Eve hadn’t….
So, this is my hope for the BI Tool of the Future, and, because I’m an Optimist, my hopes are the same as my predictions: we need a Memo-Generating Machine.
A Memo-Generating Machine does the following:
Within a certain class of questions, it translates the details of a questioner’s ask into code that returns an answer.
The answer is a memo, explaining how we answered the question, why we did it that way, and what the analysis shows.
I’d been toying with how to make the mystic machine real for a while, and then, suddenly, it became commonplace. In 2019-2022, I wrote a bit about this version of BI, and made a Rails app that implemented a rough version of the idea. It worked okay, but you had to do a decent amount of the memo generation yourself as the analyst, which made it difficult to automate for more open-ended analyses. So, it didn’t really solve the problem very well. But things have changed since those dark days…
You’ll notice, of course, that the Memo-Generating Machine is more or less Claude Cowork with well-specified instructions.
Claude doesn’t generate dashboards when you ask it to find the answer to some question you have. It writes memos, the standard output of data analysis, like a good little robot.
So, I dig what LLMs have brought to the data analysis scene because, despite their flaws, they have at least identified the correct form of the output, which is the most important thing.
The one downside of LLMs is that they’ve enabled a lot of truly shitty data analysis, and, in typical tech fashion, they have done so at scale.
The problem is they do not put structure on the form of the analysis. Often, they get some context on the database structure and how to run queries, and then they just kind of react to prompts, generating queries—often incorrectly—in subtle, difficult-to-detect ways, or they simply answer the wrong question. A lot of the folks who want to run self-service analyses can’t really tell when the LLM does something wrong or silly because they need more context about the database or statistics, etc, so it’s not really a self-service structure on its own.
You need a lot of context to do a strong analysis, and it’s unlikely to be provided by a generic solution, like a metadata layer on top of a database, because it’s not just the data that matters but how to analyze it.
So, a raw conversation with an LLM merely backloads the usual ask to the analysis team. Instead of “can you pull this?” the question becomes “can you check this?” Which is an improvement, but it’s not really self-service.
The good news is that it doesn’t have to be this way. One structure I’ve been experimenting with is being heavily prescriptive about the form of the analysis but lightly prescriptive about the form of the inputs.
Something like the following structure for, say, a Claude skill:
Tell the little robot to extract some info from the user’s prompt…
Tell it to run this exact script to do the analysis, filling in the parameter gaps as needed from the prompt. Here, I insert an exact R/Python/SQL/Shell script that does whatever analysis I’d do for this class of question directly into the skill.
Tell it to write a report on the analysis, focusing on …, noting …, recognizing that X is true, and giving it instructions for interpreting things, etc.
Feed it some of my artisanal, handcrafted memos and tell it to write like me, a human, not like it, a robot.
Step 4 is still a very much unsolved part of this version of self-serve analysis. Steps 1-3, the status quo robot does reasonably well. It does a great job of extracting parameters from natural-language input and asking follow-up questions if the user didn’t include some relevant info. It always follows instructions to run the exact script. It mostly does a good job of including the relevant constraints on the memo, but oh boy, does LLM writing suck.
It’s horrible. Worse than lifeless. I wish it were what we used to call “robotic.” Unnatural, overly stiff writing would be so much preferable to its imitation of a high school book report. Insufferable adjectives. Why does it insert imagery, metaphor, and simile in contexts where they make no sense? And so much of it. So much of it. They write, and they write. I think it’s because Anthropic charges by the word.
Things haven’t changed since Dickens*. Everyone responds to incentives. Great writers and robots alike.
* Dickens was not actually paid by the word. His tokens were “installments,” which had to be roughly 32 pages. Still, naturally, he wrote to generate installments.
AI-generated text exhausts the reader.
I have faith we’ll fix this, and, in the meantime, this version of the Memo-Generating Machine works. Highly recommended.
Aside from the writing in the memo itself, the other problem to solve with the current Memo-Generating Machine is cost. Running an analysis now has a variable cost beyond the querying and compute costs, and it can add up. Dashboards do have a significant advantage here because they’ll usually materialize a table once and run various, relatively cheap transformations on this dataset. The Memo-Generating Machine does much more analysis on the fly, so it’s more expensive (but more flexible), even before accounting for token costs.
That said, the more tightly constrained analysis setup above greatly reduces costs compared to having the LLM create its own queries or whatever. They tend to use a lot of compute, process tons of data, and burn tokens while the robot backs into something approaching a reasonable analysis via an expensive iteration process.
That’s basically what I’ve found works for cost control. If a skill is too expensive, constrain it and allow less discretion.
The “script-constrained” LLM doesn’t use nearly as many tokens, though, as the standard vague prompt to an LLM with access to your data warehouse, so it’s dramatically cheaper than that increasingly popular option.
Anyway, this is where I think BI tools are headed. Away from dashboards and towards memos. Which was always the right direction for things to go. So… good news! There are still problems to solve to make it work swell, but I’d guess this structure is the standard form of self-service analysis in the next five years.
/remind-me 2031-07-10
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn