I'd imagine figuring out that "more to it" is the big leap that would satisfy the "LLM is not AI" people. Probability plays a lot into our decision making, but there is a lot more going on in our brains than that.

I'm still hoping that Neal Stephenson was right that they are also quantum connectors to every other versions of our brains through dimensions. That'd be cool