Scott Alexander on ChatGPT as a simulator:

https://astralcodexten.substack.com/p/janus-simulators

ChatGPT is GPT-3 fine-tuned to bias toward simulating an "Assistant" character. But when it says things like "as a large language model, I am unable to love," it is not exhibiting true grounded knowledge about itself any more than when Character.AI says "As Darth Vader, I shall destroy you with the power of the dark side of the Force!"