You're absolutely right that the environmental impact depends on the source of the energy, and less obviously, by the displaced demand that now has to seek energy from less clean sources. Ideally we should have lots of clean energy, but unfortunately we often don't, and even when AI uses clean sources, they're often just forcing preexisting load elsewhere. If we can start investing in power infrastructure projects at the national (or state/province level) then maybe it wouldn't be so bad, but it never happens at a scale that we need.

I think the argument isn't the environmental impact alone, it's the judgement about the net benefit of both the environmental impact and the product produced. I think the statement is "we spent all this power, and for what? Some cats with tits and an absolutely destroyed labour market. Not worth the cost"
Especially because it's a cost that the users of AI are forcing everyone to pay. Privatize profits, socialize losses, and all that.