They can't "figure out" the hallucination thing because literally everything that comes out of an LLMbecile is hallucination. It's just that sometimes (and not as many times as people seem to think!) the hallucination happens to match something in reality.

So why do so many people believe that LLMs "know" things and aren't constantly hallucinating? We're getting our cognition hacked.

See, we have no way to directly detect intelligence in other beings. What we have are proxies that have, over millions of years of evolution, proved "good enough" a substitute for the actual measurement to be usable as that measurement. And one of the biggest pieces of proxied intellectual detection we have is ... fluency. (You know all those bigoted "Polish jokes" or, from my childhood, "Ukrainian jokes", or such over the years? Yep. Entire ethnic groups were painted with a broad brush as being "stupid" because they couldn't speak English well. Thank you proxied intellect measurement!)

LLMs are something unprecedented in the entire history of life on this planet: fluency paired with zero intellect. But we've evolved to use fluency as a very strong proxy measure of intellect. LLMs, with their confident wording and their almost supernatural fluency fool us into thinking they're, well, thinking. And because we assume that they're smart, we overlook their errors—often helping them along and prompting them down the right path when they hallucinate incorrectly, but close to correct—and when we give them the answer and they state it with fluency and confidence, we're amazed at how smart they are!

But the truth is that they cannot ever figure out the hallucination thing. It would take a completely different technology from LLMs to solve that problem. LLMs are a dead end for genuine machine intellect.