it couldn’t even begin to make sense of the question “has this poem feline or canine qualities”

Which is obviously false, as a quick try will show. Poems are just language and LLMs understand that very well. That LLMs don't have any idea how cats actually look like or move, beyond what they can gather from text books, is irrelevant here, they aren't tasked with painting a picture (which the upcoming multi-modal models can do anyway).

Now there can of course be problems that can be expressed in language, but not solve in the realm of language. But I find those to be incredible rare, rare enough that I never really seen a good example. ChatGPT captures an enormous amount of knowledge about the world, and humans have written about a lot of stuff. Coming up with questions that would be trivial to answer for any human, but impossible for ChatGPT is quite tricky.

And that’s the tip of the iceberg.

Have you actually ever actually seen an iceberg or just read about them?

It comes with rules how to learn, it doesn’t come with rules enabling it to learn how to learn

ChatGPT doesn't learn. It's a completely static model that doesn't change. All the learning happened in a separate step back when it was created, it doesn't happen when you interact with it. That illusion comes from the text prompt, which includes both your text as well as its output, getting feed into the model as input. But outside that text prompt, it's just static.

“oh I’m not sure about this so let me mull it over”.

That's because it fundamentally can't mull it over. It's a feed forward neural network, meaning everything that goes in on one side comes out on the other in a fixed amount of time. It can't do loops by itself. It has no hidden internal monologue. The only dynamic part is the prompt, which is also why its ability to problem solve improves quite a bit when you require it to do the steps individually instead of just presenting the answer, as that allows the prompt to be it's "internal monologue".

Do you truly believe humans are simply mechanistic processes that when you ask them a question, a cascade of mathematics occurs and they spit out an output? People actually have an internal reality.

Those two things can be true at the same time.

I find it absolutely mystifying you claim you’ve studied this when you so confidently analogize humans and LLMs when they truly are nothing alike.

"Nothing alike" is kinda harsh, we do have about as much in common with ChatGPT as we have with flies purpose-bred to fly left or right when exposed to certain stimuli.

People actually have an internal reality.

So do LLMs.

Can an LLM do even something that simple?

Ask it about any NSFW topic and it will refuse.

analogize humans and LLMs when they truly are nothing alike.

They seem way more similar than different. The part were they are different trivially follow from the LLMs architecture (e.g. LLMs are static, tokenizing makes character-based problems difficult, memory is limited to the prompt, no interaction with the external world, no vision, no hearing, ...) and most of that can be overcome by extending the model, e.g. multi-model models with vision and hearing are on their way, DeepMind is working on models that interact with the real world, etc. This is all coming and coming fast.

no cognizance, no agency, and no thought

Define your terms. And explain why any of them matter for producing valid and "intelligent" responses to questions.

Do you truly believe humans are simply mechanistic processes that when you ask them a question, a cascade of mathematics occurs and they spit out an output?

Why are you so confident they aren't? Do you believe in a soul or some other ephemeral entity that wouldn't leave us as a biological machine?

People actually have an internal reality. For example, they could refuse to answer your question! Can an LLM do even something that simple?

Define your terms. And again, why is that a requirement for intelligence? Most of the things we do each day don't involve conscious internal planning and reasoning. We simply act and if asked will generate justifications and reasoning after the fact.

It's not that I'm claiming LLMs = humans, I'm saying you're throwing out all these fuzzy concepts as if they're essential features lacking in LLMs to explain their failures in some question answering as something other than just a data problem. Many people want to believe in human intellectual specialness, and more recently people are scared of losing their jobs to AI, so there's always a kneejerk reaction to redefine intelligence whenever an animal or machine is discovered to have surpassed the previous threshold. Your thresholds are facets of the mind that you both don't define, have no means to recognize (I assume your consciousness, but I cannot test it), and have not explained why they're important for fact rather than BS generation.

How the brain works and what's important for various capabilities is not a well understood subject, and many of these seemingly essential features are not really testable or comparable between people and sometimes just don't exist in people, either due to brain damage or a simple quirk in their development. The people with these conditions (and a host of other psychological anomalies) seem to function just fine and would not be considered unthinking. They can certainly answer (and get wrong) questions.