You're right that there's orders of magnitude difference, but its the driving that's far more! One query to a chatGPT type model uses roughly 1Wh of energy, which is about the same as is released in burning one droplet of gasoline.