Right, gotcha. So, a few things here:

Reasonable, though it could be argued they are full of subjective experiences, just all mixed together.

This argument wouldn’t really make sense to me, because an experience and the words describing said experience are clearly not the same. An LLM only possesses the later, but the actual experience is something conscious that an LLM does not have.

Words are approximations, and often poor ones at that. For us humans, we can use them to describe experiences so that hopefully they bring understanding in the recipient, but whether the recipient truly understands and whether they can relate to the experience, also depends on their own experiences and their empathic abilities. None of that is at work inside an LLM.

But indeed, thinking that an LLM does have such abilities would be an example of attributing person-like qualities to it.

Now, if a person does develop a sense of faith from LLM output, is that more or less valid than them developing a sense of faith through a thunderstorm? That depends on the faith, I guess.

If someone truly believes the LLM to be person-like (or godlike) that would be a nonsensical faith, as much as someone believing that firestorms are the wrath of god, for instance.

If someone believes that an LLM-generated text provided them with a critical insight that turned their life around, and therefore LLMs are divinely inspired, it’s as harmless as someone believing that a thunderbolt that struck nearby gave them the necessary wakeup call that turned their life around, and therefore thunderstorms are guided by the divine.

Of course in either case, if they start to revere LLMs/thunderstorms above other aspects of reality, it becomes problematic.

Personally, as a more spiritually-inclined person, I think we should be able to say that both thunderstorms and LLMs are awe-inspiring, and I think both can, in theory, be a vessel for the divine. But ultimately, neither are divine in and of themselves.

So isn’t there a difference between faith based on LLMs vs. faith based on firestorms? I think it really depends on the case, but there’s still an important distinction: A firestorm doesn’t pretend to be human, or doesn’t try to trick people into pretending it has humanlike qualities or experiences. So generally speaking, people tend not to develop weird faiths based on thunderstorms. But LLMs are, by design, imposters pretending to have humanlike qualities. This is why I think we tend to be dismissive of people believing in things told by LLMs, because it’s so easy to fall into the idea of ascribing real humanlike qualities to them and developing psychotic thoughts from there.