It doesn't matter that there is no literal number in your brain and that there are instead chemical/electronic impulses. There is an impulse there signifying your childhood phone number. You did (and do) know that. And other things too presumably.

While our brains are not perfectly efficient, we can and do actually store information in them. Information that we can judge as correct or incorrect; true or false; extant or nonexistent.

LLMs don't know anything and never knew anything. Their responses are mathematical models of word likelihood.

They don't understand English. They don't know what reality is like or what a phone number represents. If they get your phone number wrong, it isn't because they "misremembered" or because they're "uncertain." It's because it is literally incapable of retaining a fact. The phone number you asked it for is part of a mathematical model now, and it will return the output of that model, not the correct phone number.

Conversely, even if you get your phone number wrong, it isn't because you didn't know it. It's because memory is imperfect and degrades over time.