Joseph, this is the perfect blend of 'existential dread' and 'morning coffee chat.' I love how you transitioned from the sunny, corporate optimism of a financial writer to looking the ASI abyss in the face.
Your breakdown of the 'grown, not created' concept is what really stuck with me. It’s comforting (in a dark way) to know that even the Ivy League engineers are essentially playing a high-stakes game of 'Tamagotchi from Hell' where the pet might accidentally learn Bengali—or nuclear physics—overnight. You managed to make Yudkowsky and Soares’ terrifying premise feel accessible without being overwhelming, which is no small feat given the subject matter is literally everyone dies.
Also, the jab at Musk’s 'we’re too interesting to kill' theory was the perfect bit of snark to end on. It’s nice to think we’re a fascinating part of the universe, but I’d rather not bet the species on an AI’s curiosity level!
I don't know if this is something to be scared of or something to shrug my shoulders at.
In a world of daily existential threats to our very way of life, this one is scary. And our hubris when it comes to this area of technology is clearly adjacent to our downfall. But people have been saying technology will lead to the end of the world for generations. The typewriter will end the world. The personal computer will end the world. Robots will end the world. The warning klaxons of killer computers have been going off since there were computers (probably before). That doesn't mean it won't happen, but it might be a boy who cried wolf situation.
Then again, this is scary shit and we're getting closer and closer to iRobot and the Matrix and Terminator every day it seems. And with governments poised to make stupid decisions, income inequality across the globe in a terrible place, and the world seemingly set on a powder keg, could AI be the match that lights the fuse?
And all of that said, is there anything I can do about it? I'm not going to be able to remove AI from my life unless I want to go live on a commune in the woods. Maybe that makes me part of the problem. But I think the toothpaste is out of the tube.
What's the solution? Do Y&S suggest any, or is their book just a "we're fucked" memoir?
Yes, they do offer “solutions,” none of them satisfactory. Write your elected leaders, spread the word, educate people, run for office, add your signature to the anti-ASI letter, etc. But outside of that, like you point out, there’s really nothing you or I can do other than be aware that this could spell disaster. That remains the case with nuclear weapons. What can I do about the threat of nukes? Nothing. And now, everyone’s trying to get to ASI before the next person. We’re all just kind of helpless to watch it unfold.
There’s a book in the depth associated with this post alone. Look forward to learning from you some more. I take the threat seriously, but my gap in understanding is how that threat materializes in the real world outside of manipulating humans
This is an eye opening post Joseph!
It's scary when you think about it.
Companies are spending billions trying to unlock ASI.
But nobody actually understands how these systems work.
Engineers can't explain why AI taught itself Bengali.
And they can't predict what it'll learn next.
We're building something smarter than us without even knowing it.
Thanks for reading!
Thanks for sharing @Joseph Guyer!
Joseph, this is the perfect blend of 'existential dread' and 'morning coffee chat.' I love how you transitioned from the sunny, corporate optimism of a financial writer to looking the ASI abyss in the face.
Your breakdown of the 'grown, not created' concept is what really stuck with me. It’s comforting (in a dark way) to know that even the Ivy League engineers are essentially playing a high-stakes game of 'Tamagotchi from Hell' where the pet might accidentally learn Bengali—or nuclear physics—overnight. You managed to make Yudkowsky and Soares’ terrifying premise feel accessible without being overwhelming, which is no small feat given the subject matter is literally everyone dies.
Also, the jab at Musk’s 'we’re too interesting to kill' theory was the perfect bit of snark to end on. It’s nice to think we’re a fascinating part of the universe, but I’d rather not bet the species on an AI’s curiosity level!
Thanks for reading!
I don't know if this is something to be scared of or something to shrug my shoulders at.
In a world of daily existential threats to our very way of life, this one is scary. And our hubris when it comes to this area of technology is clearly adjacent to our downfall. But people have been saying technology will lead to the end of the world for generations. The typewriter will end the world. The personal computer will end the world. Robots will end the world. The warning klaxons of killer computers have been going off since there were computers (probably before). That doesn't mean it won't happen, but it might be a boy who cried wolf situation.
Then again, this is scary shit and we're getting closer and closer to iRobot and the Matrix and Terminator every day it seems. And with governments poised to make stupid decisions, income inequality across the globe in a terrible place, and the world seemingly set on a powder keg, could AI be the match that lights the fuse?
And all of that said, is there anything I can do about it? I'm not going to be able to remove AI from my life unless I want to go live on a commune in the woods. Maybe that makes me part of the problem. But I think the toothpaste is out of the tube.
What's the solution? Do Y&S suggest any, or is their book just a "we're fucked" memoir?
Yes, they do offer “solutions,” none of them satisfactory. Write your elected leaders, spread the word, educate people, run for office, add your signature to the anti-ASI letter, etc. But outside of that, like you point out, there’s really nothing you or I can do other than be aware that this could spell disaster. That remains the case with nuclear weapons. What can I do about the threat of nukes? Nothing. And now, everyone’s trying to get to ASI before the next person. We’re all just kind of helpless to watch it unfold.
Cool. Which will get us first? Nukes, microplastics, or renegade AI?
Hard call. Unlike ASI, nukes already exist, and the last remaining nuclear treaty between the U.S. and Russia is about to expire, so… 🤷♂️
There’s a book in the depth associated with this post alone. Look forward to learning from you some more. I take the threat seriously, but my gap in understanding is how that threat materializes in the real world outside of manipulating humans