The only way to know if LLM output is accurate is to know what an accurate output should look like, and if you know that, you don't need an LLM. If you don't know what an accurate output should look like, an LLM is equally likely to confidently lie to you as it is to help you, making you dumber the more you use it. The only other situation is if you know what an accurate output should look like, but you want an inaccurate one, which is a bad thing to encourage.
"Demonstrably useful" is a lie. It's a blatant and obvious lie. LLMs are so actively detrimental to their users, and society as a whole, that calling them useless is being generous. And even if they were the most beneficial thing on the planet, there is still no reason to use the billionaire's toxic Nazi plagiarism machine.