Because an LLM’s goal isn’t to always be the most correct at answering questions. It just says what it thinks you want it to say. It’s not that telling it that it’s an expert necessarily makes it smarter, you’re just specifying not to give you an answer as though it was an amateur, which otherwise it wouldn’t have any reason not to do.

I use ChatGPT for math tutoring occasionally and when I started using the prompt “Suppose you are a professional mathematician,” I got fewer responses resembling those you might get from a classmate and more which were thorough and rigorous.