@WomanCorn Thanks -- I look forward to reading it! The best arguments so far have been coming from those who appear to understand the problem and take it seriously but expect things to play out differently.

@flats it looks like I won't have time to write a real post anytime soon, so I'll point you to this short summary instead:

https://twitter.com/WomanCorn/status/1631696104403107844?s=19

@WomanCorn The "create a good AI first to prevent the bad ones" idea always did seem kind of crazy, but (1) that a proposed solution seems crazy doesn't mean the problem isn't real, (2) I've learned the hard way that "seems kind of crazy" isn't a great heuristic to rely on in this space and (3) in 20 years nobody seems to have proposed any better idea (that seems like it'd work, not that one does either tbh)

@flats I think the problem is that a lot of their thinking on AI has a presumed final step <then we give it control over everything and it instantiates heaven on earth> and a lot of the threats hinge on the implicit assumption that you will give the AI control over everything.

So, an AI might conceal its real goals... Is that an issue if it is only going to get enough power to run the factory?

Maybe, maybe not. But we have to check every argument.

@flats If the AI isn't going to acquire godlike power, how many of the issues devolve into the principal-agent problem?

But no one wants to double check 1000 pages of blog posts to see if the conclusion relies on an unstated assumption.

@WomanCorn I haven't been privy to any of the strategic discussions, don't have a strong opinion about the "pivotal act" solutioning, and won't try to defend it (since I don't even know what the latest version of that plan might even be), but I will note that the "give it control over everything" framing sounds like CEV, which I understand has not been a live proposal for a really long time.

@WomanCorn The risk isn't that someone will intentionally give an AI control over everything, as I understand it -- it is that any AGI with preferences of any kind over how the physical world is arranged will be incentivized to take control.

@flats I think the instrumental convergence argument is still pretty good. It does rely somewhat on the idea that the AI will be trained to optimize a single metric.

When reinforcement learning seemed like the winning technique, this was a big risk. Now that LLMs are the most promising technique, it's less clear. <Minimize next token prediction error> doesn't obviously call for conquering the universe.

@flats right. The question is how many of the fundamental arguments were worked out assuming that the goal was to build a CEV sovereign and never rechecked to see if they still apply now that that goal has been abandoned.

@WomanCorn Ahh I see what you are saying. Interesting point. I will have to think about this some more.