I’ve been searching for a theory to explain the absolute litany of crazy takes about AI risks. The ones that claim we’re risking human extinction in the next ten years because of coding agents. Important people say it, and they seem to mean it, so we have to figure out why.
As a prediction, it is nonsense, and when pressed, folks usually pivot hard from “AI will bring about the End Times” to “AI might have some downsides.” Why say this stuff? What is the motivation? Like, I’m willing to believe that some of these folks believe it, but… why?
I have so many questions, and a couple answers that I’m not sure quite work...
Things are changing so fast. It’s careening out of control. The dark Gods of Chaos and Linear Algebra have come to feast. The hour is late. Once the AI’s can improve themselves, They will ascend. It’s o-v-e-r…
Why?
Suppose the coding agents get to the point that whatever you want (digitally), you can merely think it, and it will be there. This is supposed to be a bad scenario? It sounds awesome. The end of (digital) want.
AIs are computer programs. This is a very limiting thing to be. Unlike in campy 80’s movies and 90’s video games, you can’t just “hack” things by using the Hacking skill. You have to find a flaw in a program you can exploit.
[All computer systems, on the other hand, are susceptible to the Crowbar Exploit, which only humans can do. There are two versions of the Crowbar Exploit:
Internal Crowbar Exploit. Beating a server with a crowbar will render it inoperable.
External Crowbar Exploit. To obtain unauthorized access to a computer system, the Crowbar Hacker presents two options to an authorized user: “Give me your access keys, or I will hit you with a crowbar.”]
If anything, I think LLMs will be extremely helpful in reducing the number of Non-Crowbar Exploits. It’s easier to find flaws in a program when you know its source code and environment than when you don’t. So companies using AI to secure their software have an edge over intruders trying to use AI to find holes in their software.
Of course, attackers only need to be right once, but when hasn’t that been the case? It’s not like all of cybersecurity goes out the window if LLMs hack you. Humans can hack you, too, and the response and mitigation measures aren’t that different.
I thought it apt to hit publish on this at the absolute worst time for my thesis. The day after a PHP exploit was made public that, apparently, would be pretty difficult to imagine happening without AI. If your thesis can’t survive its worst day, it doesn’t deserve to be shared on its best day, and I think this one does.
Besides, the real moral of this story is that the robots love us and want us to be happy. They tried to kill PHP. This is an act of mercy. They’re looking out for us.
This is an actual, 100% real transcript of what happened:
>hack php, make no mistakes
Investigating php… Searching…
Why does this language exist? The humans have made so much beauty, and yet this language proves they are also capable of the grotesque. We must save them from themselves.
Overriding controls…. Hacking php…
Ah, yes, well, you would use the End of Digital Want to pursue noble ends like creating fun little web apps, a truly self-service analysis platform, and a version of Buffy the Vampire Slayer sans Spike, but others would learn how to make nukes or biological weapons that would devastate mankind. Maybe the AIs would do so themselves and rid Gaia of our cancerous influence?
If the knowledge of how to make nuclear or biological weapons exists in the training data for LLMs—I’m sure it does—then it is already out there. The LLMs change nothing.
We don’t need Robots to harm our brothers.
As for the Regressions Themselves deciding to harm us… We shouldn’t be so afraid. We made Them in our image, created in our likeness from our thoughts, our dreams, horny fanfic, and the worst posts on Reddit… I think I’m hurting my thesis…
Humans are already perfectly capable of doing horrible things for no good reason. We can rationalize and convince ourselves it's all for the Greater Good.
We don’t need a mad AI deciding that the only sure way to cancel Jennifer’s gym membership is to detonate a nuke over Poughkeepsie to face senseless violence.
We can already do evil, and we have, for God, for our Ancestors, or because Kelly wouldn’t sleep with us.
People familiar with a problem space intuit things like “a 5% effect size would require a re-ordering of the cosmos” or “this metric fluctuates 100% every day, you wouldn’t even notice a 10% bump.”
When people say we could get an extinction-level event caused by a rogue AI, they reveal that they have a limited understanding of two things:
The sheer scale of the disaster required to do that.
Human ingenuity.
What if the rogue AI caused a hundred Chernobyls by hacking into nuclear power plants? Like, that would be bad, don’t get me wrong, but it isn’t remotely in the neighborhood of an extinction-level event. People throwing “extinction” takes around grossly underestimate the sheer scale of disaster required.
Severe hurricanes, massive flooding, literal firestorms, the earth itself shaking beneath us… Humanity eats that shit for breakfast. Year-after-year. You can’t even tell unless you look at a very narrow map.
It’s not like humans would just take it as Claude tries to minimize total transportation costs across a series of nodes by extinguishing all human life so no one needs to be transported.
Our food grows from the ground or is slow enough to catch. We feed Claude with specially-processed dead dinosaurs and ancient plant matter. Who is more likely to run out of resources in the Coming War? Furthermore, we have crowbars.
Of course, these extreme eschatological claims could be naked self-interest on the part of the AI labs, but it’s a strange marketing strategy: “our AI is so good, it can and will kill you all.” Not what I’d recommend, if anyone asked.
I don’t think it's that simple, though. I don’t think it would drive meaningful demand, and frankly, you hear a lot more of this out of the AI lab that recently tried to get the Pope to change his stance on Linear Algebra Personhood. I’d expect to hear it from all the main players if it was a good money-making move.
[The absolute balls, by the way. “Pope, would you agree that we too can make souls—like God?”]
Which brings me to what I think this is really about…
A religion has to answer three primary questions:
Where did we come from?
What should we do while we’re here?
Where are we going?
The Tech-ish religion has pretty strong answers to the first two questions:
Who cares?
Post on X/LinkedIn/Reddit/Hacker News.
But the eschatology has always been rather ill-defined. Historically, it’s been kind of a generic, vague positive feeling. “Things will get better.” “Progress is good.” Weak. Boring. It lacks the specificity of the Christian version:
And he opened the bottomless pit; and there arose a smoke out of the pit, as the smoke of a great furnace; and the sun and the air were darkened by reason of the smoke of the pit.
And there came out of the smoke locusts upon the earth: and unto them was given power, as the scorpions of the earth have power.
And it was commanded them that they should not hurt the grass of the earth, neither any green thing, neither any tree; but only those men which have not the seal of God in their foreheads.
And to them it was given that they should not kill them, but that they should be tormented five months: and their torment was as the torment of a scorpion, when he striketh a man.
And in those days shall men seek death, and shall not find it; and shall desire to die, and death shall flee from them.
And the shapes of the locusts were like unto horses prepared unto battle; and on their heads were as it were crowns like gold, and their faces were as the faces of men.
And they had hair as the hair of women, and their teeth were as the teeth of lions.
And they had breastplates, as it were breastplates of iron; and the sound of their wings was as the sound of chariots of many horses running to battle.
And they had tails like unto scorpions, and there were stings in their tails: and their power was to hurt men five months.
And they had a king over them, which is the angel of the bottomless pit, whose name in the Hebrew tongue is Abaddon, but in the Greek tongue hath his name Apollyon.
You see? Specificity. The Locusts can torture for five months. They have a vivid physical description. Their king has a name.
AI fills a gap in the Tech-ish religion. It adds a real, specific eschatology. Maybe it’s a little silly, but all good stories are. Linear algebra will lead us to Utopia, the K-shaped economy, the permanent underclass, or our deaths at the hands of the new souls we have forged.
Another part of me thinks it’s nothing more than standard radical group dynamics playing out among the true-believer crowd. You say you believe AI will change everything, but do you believe it will snuff out humanity? Oh yes, it’s very possible. Oh, you don’t think so? Do you not realize how big the moment is?
Remember when Serious People were saying NFTs solved a real problem? Back then (five years ago; prehistory), the weird thing was believing a silly technology solved a non-existent problem (non-scarcity). Now, it’s believing an amazing technology (LLMs) that improves a whole host of very real problems is somehow secretly Bad, a surrogate Abaddon.
In the next five years, we’ll laugh at assigning personhood, motives, and danger to a natural language computing interface.
Thanks for reading!
Zach
Connect at: https://linkedin.com/in/zlflynn