But the users don't necessarily know they're interacting with "spicy autocomplete" because the companies aren't promoting and presenting it as such. They're promoting it as "your personal AI assistant" and the main way most people interact with these systems is through a chat interface. That in the background the model is being front-loaded with context and stuff gets added to the user's prompts in order to get the model to autocomplete something that looks like a transcript of a conversation is hidden, so from the user's perspective it just looks like they're having a conversation with "something".

Even for people who know in their heads how the sausage is made, the illusion might be strong enough to override that knowledge. I imagine it's kind of like when real people interact with Muppets; from what I hear, they still end up perceiving them as people, even though they can see the person with his arm up Kermit's ass.