The Oppenheimer Syndrome

The AI industry’s apocalyptic grandstanding obscures the real risks

Photo by Colin Lloyd on Unsplash

I know, you probably want to talk about bond yields. I’ll come back to the topic, I promise, but for now, what I wrote three weeks ago still stands.

---

I have always thought that the dystopian fear-mongering campaign waged by our AI gurus is largely a misguided PR effort. Greed, I believe, is the primary motivating factor for people like Sam Altman and Dario Amodei. You can see it in the way they are recalibrating their message: our technology is too dangerous, they repeat, but if only governments would kill off our potential competition with drastic regulation, then, trust us, we could make it safe.

The Madhattan Project

There is more to it, though. Altman, Amodei and others fancy themselves like the scientists at Los Alamos, pushing the boundaries of science while confronting awesome moral dilemmas.

You’ve seen the movie. (I’m sure they have.) You know the story. After watching the first atomic bomb explosion, Robert Oppenheimer quoted the Bhagavad Gita: “Now I am become death, destroyer of worlds.” Before the test, those scientists had to persuade themselves that the explosion would not ignite the atmosphere and destroy humanity, a possibility raised by Edward Teller, then ruled out as an impossibility (or at least an infinitesimally small probability…). Talk about having faith in your calculations.

Today’s AI designers follow the same script, and have raised it to the next level. With implausible sloppy precision, some claim the probability that AI will wipe us out is not infinitesimally small, but a sizable 10%. They argue that their technology will inevitably escape our control, seeding dangers we can barely imagine, let alone mitigate. And yet they feel compelled to push forward while imploring, somebody stop us, please.

It’s a ludicrous self-delusion. Today’s AI gurus have neither the scientific prowess nor the moral depth of the Los Alamos scientists. I doubt any of them has read the Bhagavad Gita in the Sanskrit original, as Oppenheimer did. But their hubris helps explain the apocalyptic scaremongering.

They’re not the same

The comparison between the physicists and mathematicians of the last century and the current breed of AI enthusiasts is stark.

• Physicists and mathematicians faced the humbling magnificence of the universe. AI developers are narcissists who believe themselves akin to gods.

• Physics and mathematics pushed our understanding of the universe. AI worships a black box.

• Physics and mathematics tackled actual hard questions. A lot of AI boils down to technical tinkering and intellectual posturing.

• Oppenheimer and his peers were forged by the hardships of war and exile. Altman, Amodei and others are shaped by the idleness of luxury.

Oppenheimer, von Neumann, Einstein and others inspired young people to sharpen their intelligence, to learn. They attracted them to science and motivated them to pursue greater discoveries. Altman, Amodei and their ilk demoralize and demotivate young people. By claiming that AI will soon surpass human intelligence across all domains and take over most jobs, they signal that there is no point in studying or working — as does their vocal support for free-lunch schemes like Universal Basic Income. Meanwhile, their products keep undermining cognition and critical thinking.

The young people who aspired to be like Oppenheimer and von Neumann wanted to be scientists and push the boundaries of knowledge; those who take as role models Altman and Amodei only aspire to make money and give moralizing lectures to the rest of us.

You guys are not serious

These differences have important real-world implications. The development of nuclear weapons brought the most serious existential risk humanity has ever faced — a risk that will only increase if, as I fear, we’re about to move into a new phase of nuclear proliferation. It was a serious danger that serious people took seriously, especially after atomic bombs were used at Hiroshima and Nagasaki. From the cold logic of mutually assured destruction to nonproliferation and arms reduction treaties, we took steps to contain the menace.

The risk of AI, by contrast, is not being taken seriously. Yes, the dominant narrative says that AI systems are already escaping human control and are well on the way to recursive self-improvement. We read it in OpEds and hear it in countless rants by callow software engineers. But if people like Altman and Amodei truly believed what they say, they would fold their companies and retire to the countryside to grow tomatoes and raise goats. If governments believed what Altman and Amodei are saying, they would shut down the companies and throw the executives in jail. This, after all, is what they would do if they heard the same claims coming out of synthetic biology labs.

A productivity killer after all…?

It is also worth noting that the AI industry is talking out of both sides of its mouth. To entice individuals and businesses to buy its products, it boasts that they are high-precision systems, able, when skillfully prompted, to solve our most mission-critical problems. But to build the hype, it warns they are self-determining and self-improving entities that can in no way be controlled or contained. Both things can’t be true. If AI systems are independent, uncontrollable entities, they are as likely to destroy your business just for fun as they are to boost the efficiency of your supply chain — and the value proposition evaporates.

I will go further. Our economies and societies are almost totally dependent on digital systems, from our financial sector to our telecommunications infrastructure, to the electricity grid, to transportation networks. If AI systems cannot be controlled, there is a high risk that they will start wreaking havoc across our economy. Our utter dependence on digital systems will emerge as an unacceptable liability and will need to be unwound — reverting to an analog economy would be the ultimate kill switch, and presumably the only one available. But if that is the case, far from bringing us marvelous productivity gains, AI will be the worst adverse productivity shock we have ever suffered — and then the whole case for developing AI disappears.

So why is the risk of AI treated differently from that of nuclear war? Partly because there is a big difference between seeing an atomic explosion and reading about rogue AI agents. Partly because aggressive entrepreneurs like Altman and Amodei are shouting louder than the more sober minds at the forefront of AI science. And partly because the caliber of today’s political leadership is, well… the less said the better…

Pity

It’s a shame. AI poses deep questions about the nature of intelligence, awareness and consciousness. The technology can deliver important efficiency gains, though, I fear, nowhere near as significant as the hype drumbeat keeps promising. And even without falling into dystopian Skynet scenarios, it does carry substantial risks. It deserves more serious debate and thoughtful handling. In the age of TikTok, this seems hard to achieve; but without it, we’ll reap much more of the downside than the upside.

We could start with a reality check: today’s AI gurus suffer from the Oppenheimer Syndrome, but they are no Oppenheimer — not by any stretch of the imagination.