The Neglected Read #4: The Book That Made Me Take the Risk of Artificial Superintelligence Seriously

IF ANYONE BUILDS IT, EVERYONE DIES, by computer scientists Eliezer Yudkowsky and Nate Soares, is a wakeup call to supporters of AI.

Photo by Joseph Guyer

During a 2023 interview with 60 Minutes’ Scott Pelley, Google CEO Sundar Pichai admitted to something that should have convinced way more people that artificial intelligence (AI)—and artificial superintelligence (ASI) specifically—poses a unique threat to humanity.

Some of Google’s AI supercomputers had recently taught themselves how to use Bengali, the official language of Bangladesh, without being prompted. Google engineers had not trained them to do this, and they were at a loss to explain why this happened.

Pichai himself had trouble accounting for these “emergent properties,” as he called them. Researchers “don’t fully understand” why some AIs are able to pick up new skills they haven’t been trained on, he said.

So why, Pelley reasonably asked, release the AI to the general public?

Pichai’s answer was just as evasive: “[I]t’s good that some of these technologies are getting out, [so] people like you and others can process what’s happening.”¹

But if genius-level, Ivy League-trained engineers are clueless to “process” the AI’s uncanny behavior, what hope does Joe Six-Pack have?

And if the AI has the unexpected ability to teach itself another language without any inputs, what’s stopping it from learning how to, I don’t know, build a nuke? Or develop a deadly new virus?

These considerations are at the heart of If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, computer scientists Eliezer Yudkowsky and Nate Soares’s eye-opening treatise on the rapidly growing dangers of artificial intelligence.

And before you ask the question, the answer is an emphatic yes: Yudkowsky and Soares literally believe that ASI will result in humanity’s total extinction.

I chose If Anyone Builds It as my second Neglected Read of 2026, even though I've owned the book for only a couple of months. I often write about the financial and economic side of AI in my role as a financial writer at an asset management firm, and after years of saying only positive things about the technology—it’ll drive the economy for years to come, make investors boatloads of money, increase productivity, and lead to advancements in health and science—I began to wonder: at what cost? I sought expert insight from sources that came at the subject from an ethical angle, and the aptly titled If Anyone Builds It, Everyone Dies seemed to fit the bill.

If Anyone Builds It…

As cofounder and president, respectively, of the Machine Intelligence Research Institute (MIRI), Yudkowsky and Soares have long been leaders in the nascent field of AI alignment, which seeks to ensure that AIs act in accordance with human values—in other words, not enslave us Matrix-style or terminate us, well, Terminator-style.

If you’ve seen the 2004 film I, Robot, starring Will Smith, you should be familiar with the idea of AI alignment. The movie is very loosely based on a series of short stories by science fiction grandmaster Isaac Asimov, who in the 1940s conceived of the Three Laws of Robotics: 1) Robots may do no harm to humans, directly or indirectly; 2) Robots must obey human commands; and 3) Robots must practice self-preservation.²

In 2023, Yudkowsky and Soares (I’ll start referring to them as Y&S for simplicity’s sake) were signatories of an open letter to the world’s leaders and decision-makers, urging them to come together and acknowledge that AI poses a serious risk to the survival of humanity. The letter consisted of a single sentence:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

ChatGPT, the world’s best-known large language model (LLM), had been publicly launched just six months earlier. But Y&S weren’t necessarily thinking about LLMs when they added their signatures to the letter. They were looking ahead to ASI, a type of super AI that’s many times smarter, faster, and more innovative than all of humanity combined. If Google’s AI managed to teach itself Bengali in 2023, then an ASI, by extension, would be able to teach itself the entirety of human knowledge… and then some.

ASI isn’t a thing—yet—but the success of ChatGPT and other AI chatbots has sparked an arms race rivaling the Manhattan Project and the Space Race. As we speak, companies around the world are breathlessly spending hundreds of billions of dollars to be the first to unlock superintelligence, an advanced technology that sci-fi writers like Asimov have been warning us about for decades. Think HAL 9000, Agent Smith, Skynet, Marvel’s Ultron.

AIs Are Grown, Not Created

Throughout the book, Y&S stress that, contrary to what some people think, AIs (including LLMs) are not created the same way an app or a video game is made; they’re grown—like people.

What does this look like in practice? I’ll be honest… I have no idea, despite Y&S spending several pages trying to describe it. I don’t feel so bad about my shortcomings, though, because—once again—it doesn’t seem as if the engineers themselves fully understand the process either, which should give us all pause.

The most fundamental part of training an AI involves a process called gradient descent. This is what’s supposed to minimize errors, fabrications, hallucinations, and other hiccups.

A helpful analogy is that gradient descent is to AIs what natural selection is to humans. Without getting technical, both forces are understood to optimize their subjects' adaptability through constant trial and error, but no one really knows how exactly. (Some people prefer to compare gradient descent to selective breeding instead of natural selection.)

Here’s the fun part: gradient descent, like natural selection, doesn’t always lead to expected outcomes. Just as a person can be born with a mutation or neurological disorder, so too can an AI occasionally behave erratically. Like learning Bengali on their own.

No doubt you’ve experienced AI errors and hallucinations yourself—often to humorous effect—while chatting with ChatGPT, Grok, or another chatbot; or maybe you’ve seen wild, uncomfortable AI-generated videos that ignore the laws of physics.

Remember when Health and Human Services (HHS) Secretary Robert Kennedy Jr. released a long-awaited report on childhood health that contained numerous fake studies and citations? Blame Kennedy’s staff for using an LLM to write an important government report in the first place, but the incident should make it clear that gradient descent, like natural selection, is not an infallible process.

The Myth of AI Alignment?

Compared to what could go wrong, learning Bengali and misleading parents with made-up health studies is pretty tame and innocuous.

What happens when a super-advanced AI in the year 2032 starts showing strange, erratic preferences that engineers didn’t account for (surprise!) and that are not in alignment with human values or, indeed, human survival?

“Unplug it” isn’t an option because by the time we noticed something’s awry, it would probably be too late. Y&S treat the reader to a number of juicy imagined scenarios of how a network of advanced superintelligences could easily trigger the destruction of humanity, yet the most terrifying possibility likely remains beyond our current imagination.

“The preferences that wind up in a mature AI,” Y&S warn, “are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained.”

To reiterate: AIs are grown, not created, and the process of growing them is arcane at best. We can’t just “program” them to treat humans benevolently at all times the same way we program Mario to jump when we tap the B button. It would be impossible for a geneticist to tell you if a fetus will grow up to be the next Hitler merely by examining its DNA, and similarly, no human being on earth can look at the trillions of parameters and weights that make up an AI’s infrastructure and determine if it will be “safe” or not. As I write this, the Wikipedia page for Deaths linked to chatbots has 32 references, and there’s no doubt in my mind that this number will rise exponentially as AIs get more sophisticated and persuasive.

…Everyone Dies

Knowing all this, we should be extraordinarily skeptical when the executive of an AI company tries to make the claim that their platform can be trusted.

Near the end of their book, Yudkowsky and Soares mention Elon Musk, whose company, xAI, runs the Grok chatbot. Before Grok came onto the scene, Musk gave an interview in which he said his AI would be benevolent to humans because it would be designed to “understand the nature of the universe.” Such an AI, he claimed, is “unlikely to annihilate humans because we are an interesting part of the universe.”

With all due disrespect to Musk, his adolescent comment ignores everything we’ve talked about. Now, I agree with Y&S that Musk’s rockets are to be greatly admired, but let’s not forget that many of them have exploded after launch. This is to be expected in the trial-and-error phase of advanced rocketry (rapid iteration); in the world of ASI, however, a failure rate of this magnitude would surely spell the end of humanity.

---

My Next Neglected Red

For my next Neglected Read, in observation of Black History Month, I’ll be checking out The 1619 Project. Happy reading!

1. A clip of the interview is available here.

2. You can read Asimov’s short story “Runaround,” which contains the first instance of the Three Laws of Robotics, here.