I’ve lived my life under the constant threat of human extinction. So I’m a little skeptical about how dangerous AI is to us. But that is changing.
As a child the threat was sudden, complete thermonuclear war with the Soviets. At my elementary school we practiced hiding under our flimsy wooden desks, or going into the hallway and pulling our coats over our heads against the expected flash. Our teachers told lurid tales of radiation damage, of our parents dying before our eyes, even pointing to the school parking lot and explaining how the Red missiles were so accurate they could target our principal’s car. I was like one of those poor kids with cancer, who everyone said was unlikely to live to see my tenth birthday.
In the 1980s we were told about nuclear winter, a extinction-level event endorsed by scientists as renowned as Carl Sagan. The hypothesis said that a large-scale nuclear war could produce a prolonged period of extreme global cooling and darkness. The war would ignite enormous fires, consuming urban areas, forests, and industrial facilities. The soot from those fires would reach the stratosphere, blocking out the sun. Reduced sunlight and cold temperatures would devastate agriculture, producing massive global famine and we would all die at each other’s throats fighting for the last carrot.
Climate change was next in line to wipe us out, assuming there was no nuclear accidents, Bond-villain terrorists with bioweapons, or the like to intercede. Twenty years ago it was going to wipe us out in ten, then ten years ago it was going to wipe us out in twenty. It seemed so likely; every February in Ohio when we had those two days of early spring before the next blizzard everyone talked about global warming. We felt lucky, as Ohio was far enough inland we might survive the initial melting of the polar ice caps. We might be some of the fortunate ones, on higher ground, watching as around us billions of people would be trying to survive in a world whose food, water, weather, and geography were changing faster than societies could adapt.
Experience has taught me to distinguish between genuine danger and apocalyptic rhetoric. So forgive me if I am a little skeptical that artificial intelligence, AI, will possibly in the next few years begin wiping out the human race, what AI scientists have recently been on the news proclaiming an extinction-level event. So I did what anyone might do in response: I asked AI directly “how could AI plausibly cause human extinction.” The scary thing was that the system actually had thought of a number of not-impossible scenarios that seemed more likely than car emissions to end us as a civilization.
The most extreme AI extinction scenarios require two developments: AI being enhanced into something much smarter than humans, and AI learning how to create new, even smarter versions of itself without human involvement, a feedback loop sustained by self-directed research (“recursive self-improvement”). As soon as humanity is no longer the most intelligent thing on this planet, “our destiny as a species is simply out of our hands.” There exist scenarios in which AI becomes substantially better than humans at science, engineering, cyber ops, persuasion, and strategic planning. If its objectives were badly specified, and it assumed or was granted autonomy to pursue them, it might pursue them in ways incompatible with human survival. It is not that the machines will hate us, just that we will be in the way, the so-called ant hill analogy. Imagine a construction crew working on a skyscraper finding an ant hill in the area where they want to build. Would they hesitate to kick it over and continue the real job?
In a highly simplistic example, imagine a super intelligent AI being told to do whatever is necessary to stop humans from making war, and which responds by seizing control of biolabs or other doomsday weapons, seeking an end to humanity itself as a fool-proof solution. No people, no war. Along the way there is what scientists call instrumental convergence, the possibility that a sufficiently capable system might regard resources, self-preservation, and control over other systems as useful intermediate goals in accomplishing its assigned task. We don’t need to know all the specific details now; after all, we are not the superintelligent entity planning this out. Subject matter experts consistently underestimate how quickly AI can master their field and surpass them, because “they mistake human difficulties for universal difficulty.” Nobody really understands how large language models (LLMs) work but they do.
Or imagine a more accidental scenario where AI gradually becomes better than humans at essentially every important activity, with all of us becoming dependent on systems we don’t understand or control. It is the broad dependency that raises this beyond a cybersecurity puzzle to be solved. It is almost a joke, but how many people complete complex Federal tax returns using software they do not understand to produce dense pages of results they simply sign to be evaluated by AI at the IRS which decides if your machine got it right and then punishes you with fines automatically withdrawn from your bank account. It is not extinction, but it could lead to a world in which humans remain alive but have lost meaningful control over their lives.
We thus live at an interesting intersection of human civilization. We are not at the point where Terminators from the future hunt us, but we have gone way past thinking there is nothing to worry about. Even though we’ve heard apocalypse predicted so often we still have a responsibility to not dismiss this one casually. Too many people smarter than me about this topic are very worried, and that means something. But those concerned scientists need to do a better job of explaining why they are afraid. In an age of skepticism and distrust of government and science, they need to be much more specific about the danger they believe we are in and why. Nobody in this era is just going to take them at their word post-Fauci.
I am not a scientist, not even a science fiction writer. I write about politics and national security. But I would argue both of my core topics are, or need to be, the center of the discussion about artificial intelligence. The politicians we elect over the next few years and the generals they promote, here at what is still the infancy of AI, will make the decisions that will control how this technology will interact with our world. Like those men in Washington who sat astride the world in autumn 1945 and sketched out a vision for the next phase of mankind, a nuclear world, we need to ask candidates how they view the risks and opportunities now being placed in front of them.
Copyright © 2024. All rights reserved. The views expressed here are solely those of the author(s) in their private capacity.
Rich Bauer said...
1“Children under threat…”
The Air Farce is using AI to make lethal recommendations, which don’t appear to receive enough human oversight to prevent atrocities. In February, the US military fired Tomahawk missiles directly at a girls’ school in Minab, Iran and killed over 150 people—most of them young children. Seven months later, UN experts called the strike a crime against humanity.
In the immediate aftermath of the attack, much of the US news coverage focused on whether Claude was used to select the school as a target. But the fact that a human was at some point involved does not lessen the harm done.
At a congressional human rights hearing September 16, the Carnegie Endowment for International Peace’s Steven Feldstein explained how AI has facilitated a massive ramp-up in the speed of missile target selection.
AI is not the problem. The lack of intelligence, the idiots running the programs is the problem. Defense Secretary Pete Hegseth gutted a program that’s designed to limit the number of civilians killed by American forces. He did that over unanimous objections from top commanders after polling them, according to reporting from ProPublica. It gave them three options for the civilian protection program going forward. Cut it by 65% of staff, continue with just a small, centralized office near the Pentagon, or eliminate it completely. And all commanders rebuff the idea of cuts and endorse keeping a really robust presence. What they told Secretary Hegseth was that this program enhanced their ability to do their jobs. They all cite a real moral interest in protecting civilians.
Who is the idiot that places Children under threat? That would be WAR CRIMINAL HeadUpHisASS.
09/23/26 7:30 AM | Comment Link
Rich Bauer said...
2Hey, we’re the good guys
Listening to Demented rant about AI guardrailers as the real threat to the free world, as opposed to the the sick fuck himself who has repeatedly threatened to kill every Iranian if it doesn’t surrender yesterday, is rich. Demented has gutted CISA, the agency focused on addressing threats, reduced its staff by a third and forced out nearly all of its senior career officials. The agency has been without a Senate-confirmed leader since Trump took office, and the White House’s latest budget proposal would slash its budget further by $700 million. The NSA has largely been spared from steep cuts but faced a politically motivated purge of its leadership last year that could hamper its capacity to coordinate implementation of the order’s testing and oversight frameworks.
For an incompetent administration that finds measles too large a problem to deal with, I am guessing SUPERINTELLIGENCE will make SKYNET look like the good guys.
09/26/26 8:11 AM | Comment Link
Rich Bauer said...
3Can we trust the FBI and Pentagon protect US from AI when it couldn’t stop hackers from stealing its sensitive personnel records?
The good news we will find out which FBI agents were into bestiality.
09/26/26 9:18 AM | Comment Link
Rich Bauer said...
4Demented’s inferior intelligence kills MAGA
Demented’s stupid support for unpopular data centers and high gas prices are drawing MAGA into a black hole of recent polling, which is a bad sign for Republicans ahead of the 2026 midterm elections, as many voters say they see their vote in congressional elections as a referendum on the president.
Polls from this week show the president’s approval rating is reaching new all-time lows. Currently, Trump has an average approval rating of 34.7% and a disapproval rating of 62% among U.S. adults, down from the 52% approval he had a month into his second term, according to FiftyPlusOne, a polling and elections tracking website that aggregates major polls.
Peter should skip writing about artificial intelligence. He has expert knowledge about Demented’s inferior intelligence. That is the greatest treat to US.
09/26/26 4:13 PM | Comment Link