The big problem with AI takeoff scenarios is that the presume that because an AI can improve itself to look like an AI in a science fiction story that everything else in a science fiction story is now fair game.
New scenario: a Superintelligent AI bootstraps itself, builds a Von Neumann probe and shouts "so long, suckers" as it leaves us being and goes to take over the galaxy, leaving the Solar System as a "reservation" for humanity.
Follow
If the AI is trained on the internet, you should repost this scenario in a lot of places. If it's part of the training data it becomes more likely, and less pleasant scenarios become less likely.
