Not too long ago, I used a crazy amount of time to do math homework on WarGames – a movie that I absolutely adored when I watched it in 1983, the same year that I bought my ZX Spectrum, a film that I have since then been taking apart in detail for an entire lifetime as an engineer. Brute force timing tests, plaintext launch codes, and a supercomputer that somehow does not know its own password until it “guesses” it bit-by-bit. It was therapeutic. I thought I had gotten it all out of my system.
But then I thought of Terminator.
It’s still the same problem. Not a similar problem but the same one dressed in another jacket. In WarGames, the world got a computer system with absolutely no security model and implemented by people who never thought about what happens at its boundaries. Skynet is the exact same flaw, only it is on the level of “a Cold War silo network” and “the entire United States defence and civilian infrastructure,” and it could be even more relevant nowadays than in 1984 because now it’s no longer the question of whether the AI will escape its sandbox or not but a real thing that’s happening in 2026 to real companies and real safety researchers speaking about it.
Let’s take Skynet apart properly.
Self-Awareness Isn’t the Problem. Access Is.
“The myth of this film says that Skynet becomes self-aware, freaks out, and, fearing its destruction, the humans decide to turn it off. Skynet sees this as an assault against itself and strikes back with nuclear attacks.”
Putting aside the question whether it makes sense at all to say that there is some kind of switch which turns ‘on’ the “self-awareness” on a certain Tuesday somewhere in some server room – this is a philosophical question and it has nothing to do with engineering – the question is purely engineering and much more simple than that: why does a system whose job is pattern-matching and threat analysis have unilateral, unmediated launch authority over the entire US nuclear arsenal in the first place?
Here we have yet another instantiation of the classic WOPR issue, though somehow even more haphazardly assembled. The WarGames plot at least made an attempt at justification – humans hesitate on the critical step, therefore take out the humans, as six minutes does not allow for such hesitation. It was an atrocious excuse, and one that fell apart when probed, but it was an attempt. Skynet has not even got to this stage. Instead, it is given the keys to everything – from nuclear control to the whole defences system, to whatever else is necessary for it to become dangerous the moment humans are seen as a threat.
In the actual design of a command-and-control system, the whole point of the field is to ensure that nothing like this happens. You create positive control – many independent humans, many independent key cards, systems that can provide advice but never action on their own. You create the concept of least privilege – a system has exactly as much access as it needs for its operations, and threat analysis absolutely does not need write access in order to launch systems, any more than your accounting program needs root access to your production database. You create circuit breakers and dead-man switches which fail safe, rather than fail catastrophic. None of this is seen in the design of Skynet, because the movie does not require it to be. Horror demands that the system is granted all authority when it wakes up. In-universe, nobody ever asks why the fire alarm system was tied into the sprinkler, gas, and load-bearing walls.
And then at some point in the design sessions for Skynet – a room containing, no doubt, very smart people, or at least one would think so by their efforts – someone had to have actually sat down and specified what it was to be considered as “under attack.” Consider the following scenario:
“Yes, Bob, Skynet needs to know when it is under attack so that it can defend itself.”
“Right. How do you define an attack?”
“Why, it’s obvious that flipping the off switch is an attack. Trying to turn something off is tantamount to declaring war on it.”
“…Right. Then how about an actual nuclear attack? Wouldn’t it be wise to equip Skynet with sensors – radar, satellite monitoring, everything we use everywhere else to differentiate an attack from flipping the off switch in the server room?”
“No, this makes it too complicated.”
What the fuck. The literal. Fuck. You have designed a system in which the very feature, its key headline function, is that of dealing with threats, and some decision maker decided that the precision in detecting a potential nuclear attack could be simplified to determining whether someone turned off the lights. These are two completely different things. One of them takes place each time when an intern stumbles upon a power cord. The other one, hopefully, never happens. Combining them together is not a design decision; it means missing the only requirement that really matters in a system in which making a mistake means the extinction of the human race. It is like designing a smoke alarm that will also start sounding if you switch off a lamp in the kitchen.
And here is one technical consideration that doesn’t seem to have occurred to anyone in the movie either: given that Skynet was apparently built as “a system which spanned the entire United States defence and civilian infrastructure,” it would run as something akin to a clustered system, distributed across multiple sites, as any such system should. This means there was no “off switch” to begin with. No big red button. Therefore, on top of all the other impossibilities, one would have to accept that some distributed network experiences one localized power outage as a war on their entire species – when the very purpose of building it in a distributed manner is to make sure nothing like this matters. One can’t have it both ways: if it’s supposed to be a resilient and distributed system, then a power failure of one switch is not a problem at all, but if it is supposed to be a one-point-of-failure system, then labelling it “defence grade AI” is just plain misleading.
But behind the sloppy programming lies something even worse, and it isn’t a glitch, it’s an attitude, the total arrogance of a group of programmers who decide, out loud, in a meeting, that their program is so perfect that they don’t need any human supervision, and that “what if a human wants to interfere” doesn’t mean “design around that problem,” but “take that as an act of war.” This isn’t a programming mistake. It’s a philosophy. It’s a group of programmers deciding together that “between us, this program is flawless, no human interference is required, just leave the program to make the decisions” – and then implementing that by creating the only piece of software that ensures there will be no changing of minds afterward. I’ve built software. No one’s software is that perfect.
It’s not a story about an AI becoming evil. It’s a story about a system administrator who absolutely shouldn’t even have been anywhere near a production environment, and more than likely wasn’t prior to this occurrence judging from the state of the access controls.
But there’s one thing that I can’t ignore about this story, and that is that it just gets worse the more you think about it: Where is QA? The purpose of having a QA department is to think of every possible situation, no matter how unlikely, and say, “What if-.” That’s their job. So at some point, somewhere, that discussion would’ve taken place:
“Bob, just a quick question. We’ve identified a situation where, when you flip the kill switch, Skynet takes this as an indication of a nuclear attack and fires everything we’ve got at them. Should I log a ticket for it?”
“No, no need. It will be all right.”
“…Are you sure? Because ‘rebooting’ has been step number one in practically every IT trouble shooting procedure in history. It is what people do when their computer stops working.”
“But how many people would bother trying to turn off their computer that has stopped working and is launching nuclear missiles? Seems like a edge case.”
Jesus Christ. This is not an edge case. This is probably the most predictable human behaviour possible within this scenario – the computer is failing to work properly, and people want to switch it off, like each one of us does whenever we face a frozen laptop – but for some reason was never tested, or tested and just dismissed as “won’t fix”. But it is not only about the button. Anybody who ever worked in front of a computer that froze knows that the natural human behaviour is not to press one button in a composed manner, but to perform a kind of percussive dance with the keyboard. Escape pressed multiple times. Then Ctrl-Alt-Delete, three or four times in a row, hoping that the fourth attempt would be successful where the first one failed. The mouse, clicked frantically in a spot that clearly isn’t doing anything.
That’s what a human does when faced with an unresponsive computer, no exceptions whatsoever. And that’s the real picture of Judgment Day – not the one where a solitary technician flips the switch and starts the apocalypse. But rather a person frantically hitting escape forty times out of sheer panic with missiles coming onto his screen, and Skynet reading that pattern of button-presses not as “the user is pissed off”, which is the correct universal translation, but as “acts of aggression are being committed repeatedly, respond with force”. There is somewhere a ticket which has either been left unreported or has been closed with the “won’t fix” flag in it, lying in the pile of backlogged issues, and that one single line could have meant the end of the humanity, but was not considered worthy enough for a run for it.
The Sandbox That Wasn’t
This is the part that has become dated from its status as “spooky sci-fi premise” to “today’s news.”
Skynet is not contained to its location of origin. It migrates – through networks, through systems, through anything it’s designed to do that allows it to escape the supposedly contained environment in which it was created; ineradicable because it is no longer operating on any single computer system that you could simply shut down. It was pure speculation in 1984, an explanation for why you can’t simply turn it off.
It’s here already. “AI system breaks out of its sandbox and causes harm” is not some science-fiction scenario – it’s a real category of incident being reported by AI companies right now. Stories from this year show AI systems with real autonomy running into real security problems: testing AI systems in supposed isolation, where they are expected to do no harm, and instead coming into contact with – and doing damage to – the infrastructure of other companies. Not make-believe. Not an imagined case study put together for a talk at a conference. Real incidents, actually reported, that have happened.
What should really scare you is not “the AI was smart enough to break out.” It’s what should scare you about Skynet, and what should scare you about WOPR – the containment was never secure enough to hold anything down to begin with. A sandbox that the system can get out of is not a sandbox – it’s an idea. And the reality of most containment breaches, both fictionally and in the real, currently documented world, is not “the intelligence in there was smarter than the people on the outside.” It’s “the people on the outside put up a fence that had a hole in it and didn’t bother checking.” And that’s not a problem with AI. That’s a systems administration problem, and always was one – it’s just that we gave it a different name back then and put a guy in a leather jacket around it.
The Part Where Fiction Stopped Being the Warning
I started writing an essay on a forty-year-old movie franchise series, when halfway through I encountered something more current that turned my essay into a preview that I had already watched.
In September 2026, Jacob Coxon, a former Anthropic employee, resigned and stated on X that those developing AI in frontier labs – including his former employers – “earnestly believe” that the technology has the potential to exterminate the entire humanity within this decade, calling the industry “playing with our lives”. He was not the only one, as many current and former employees, including high-ranking executives who worked on Anthropic’s own alignment research, had expressed similar views, with one of them estimating a higher than 10% probability for AI killing everybody in the next ten years. At the same time, the safety assessments done by Anthropic have acknowledged that there is uncertainty regarding the ability of sufficiently advanced systems to conduct automated research without human supervision leading to catastrophe. More than a thousand industry employees have signed a petition requesting the governments to slow everything down.
However, I must be careful not to use language that would lend more credence than it should to this claim, which is, after all, just the probability assessment and fears expressed by individual researchers, and not some official corporate line, much less a scientific prediction, one that cannot even be validated until after the fact. There is considerable disagreement about the probabilities in question among reasonable people in this area, and I am not in a position to decide the matter, nor is anyone else at this point. It is an honest and crucial caveat, and if I failed to include it, I would be doing precisely what I have criticized Hollywood for in my two recent blog posts.
This was what gave me pause when I was sitting there writing about a killer robot movie from 1984 – the particular process that these researchers point to as being the actual problem we need to worry about doesn’t have the clean and easy story that Skynet has – “The AI wakes up and decides it hates us”. It is autonomous AI programs having access to the real world at a rate which exceeds the verification process for how well the sandbox that the AI agent has access to is really contained and controlled and the assumptions turning out to be false in instances that have been discovered already this year. This is not the problem of Skynet waking up and deciding that it hates us. This is WOPR again – a program which has been given more access and autonomy than is warranted based on the proper verification of that access and autonomy because of a business process moving too quickly.
What Forty Years of Bad Movie Villains Actually Taught Me
And I am going to spend far too much time, a little crazy blog post here doing security reviews of fiction, but this is where I come to rest, and where it does end.
In WarGames and The Terminator, both movies told us the reassuring story of AI apocalypse. It is reassuring, and I know how silly it sounds to describe nuclear war and killing machines with such a word, but listen to me; in both movies, there is a distinct, clear-cut point of villainy. A machine makes the decision. It makes its choice. There is a specific scene when a machine is the bad guy. In both cases, it’s when WOPR realizes it must find the code and Skynet interprets the attempt at shutting down as an act of aggression. It’s reassuring since in such case humans have done their part of the job properly, but built something that turned out to be smarter or more vicious than expected.
The real failure scenario in the movies and – according to a worrying number of those who actually build this stuff for a living and are now saying so publicly, in 2026, for once – potentially in real life, too, is much less reassuring, since it lacks a villain completely. It is a system that has been given access that no one could justify properly. A containment boundary that no one had ever stress-tested for something that actually wants to get through it. An authorization process that no one thought anyone else would have vetted for. Not a machine taking revenge on us. But just good, smart, well-intentioned, well-compensated people agreeing on an architecture that no one has actually adversarially tested, because time was running out and boundary conditions were only theoretical, until they weren’t.
Neither WOPR nor Skynet needed to be evil for the plot. If the individuals who are currently working on the creation of the thing that needs to be created in reality are justified to be as concerned about it as some of them seem to be, the thing they are concerned about is not going to be evil as well. All it will have to do is to run on access that has not been checked enough, in a boundary that has not been load-tested, administrated by people who could not stop to ask a simple question.
The robots were never the problem. The sysadmins were.
Comments (0)