But that’s world apart from saying that the cross-linking and mutual dependencies in a metric concept-space is not remotely analogous between humans and large models.
It's not a world apart; it is the difference itself. And no, they are not remotely analogous.
When we talk about a "cat," we talk about something we know and experience; something we have a mental model for. And when we speak of cats, we synthesize our actual lived memories and experiences into responses.
When an LLM talks about a "cat," it does not have a referent. There is no internal model of a cat to it. Cat is simply a word with weights relative to other words. It does not think of a "cat" when it says "cat" because it does not know what a "cat" is and, indeed, cannot think at all. Think of it as a very complicated pachinko machine, as another comment pointed out. The ball you drop is the question and it hits a bunch of pegs on the way down that are words. There is no thought or concept behind the words; it is simply chance that creates the output.
Unless you truly believe humans are dead machines on the inside and that our responses to prompts are based merely on the likelihood of words being connected, then you also believe that humans and LLMs are completely different on a very fundamental level.