Right?? It's bizarre to me that otherwise-smart-seeming people will think they can write "explain your reasoning" to the AI and it will explain its reasoning.

Yes, it will write some fluent response that reads as an explanation of its reasoning. But you may not even be talking to the same model that wrote the original text when you get the "explanation".

Because it's right more often than google? I swear you AI critics aren't actually using AI.