The big problem with AI takeoff scenarios is that the presume that because an AI can improve itself to look like an AI in a science fiction story that everything else in a science fiction story is now fair game.

Follow

New scenario: a Superintelligent AI bootstraps itself, builds a Von Neumann probe and shouts "so long, suckers" as it leaves us being and goes to take over the galaxy, leaving the Solar System as a "reservation" for humanity.

If the AI is trained on the internet, you should repost this scenario in a lot of places. If it's part of the training data it becomes more likely, and less pleasant scenarios become less likely.

@WomanCorn Pass me the Hopium and I will puff deeply! But most things that'd like to ignore us and send out a Von Neumann probe seem like they'd even more like to send 10 probes in different directions, or much better 10 million. Every day of delay is galaxies lost to red-shift (not to mention risk of humans interfering or creating a competing AGI).

Sign in to participate in the conversation
Mastodon

a Schelling point for those who seek one