I've been up for 28 hours now so I might have missed something but:

That's a preprint, not peer reviewed or as far as I can tell really even discussed anywhere. Already not a strong paper to base your argument on.

The self awareness does not seem to be meant as conscious self awareness but rather "it keeps stored data and will use that as a bias".

I'll pull one example from the paper but I could pick apart the whole paper in a similar fashion (pick apart your saying it hints at consciousness. The paper itself seems solid in the context of "we can look at these encoded bits of information to detect hallucination"). "The LM encodes a notion of truth (and false) as linear directions within its representations" that's that vector mathematics I was talking about. directionality in multi dimensional space and the vector between them is how they encode information.

"we show that LMs encode meta-knowledge (i.e., ability or inability to correctly recall) about the fine-grained factual relationships as linear directions" linear direction? Hmmm what's a vector again?

What you're seeing is a very common problem with communication in general. We use the same words to mean so many different things. It's especially bad with AI because they were intended to mimic human brains as we understood them in the 40s. That meant we pulled a lot of terminology from psychology and biology and ended up mixing them all in a hodgepodge with statistics terminology. It's a problem we've yet to sort and is clouded by people coming into the space who don't understand the underlying tech or the context in which the language evolved in this space.

If I had to guess that's why the author italicized "hallucination" and "forgetfulness" in the introduction. And added the word "insinuates". Because the machine isn't actually forgetting or hallucinating because it isn't actually conscious and they wanted to emphasize that.