I'm just so dumbfounded that this isn't obvious to everyone who has 1. average intelligence, 2. a five minute explanation of how it works.
You should trust it exactly as much as a magic 8 ball. Alternatively, replace all source reference of "according to << favorite packaged LLM >> ..." with "according to my 10 year old nephew who is playing a game of never-say-you-don't-know...".
Which isn't to say that LLMs can't be useful. But if you trust any fact based output from such a text generator, that you can't (or don't) verify yourself, you seem exactly as dumb and liable as if you said "but... but... the magic 8 ball said it would be fine!".