I honestly assume (given my experience with the precursors to LLM's) that this is because there are humans on the other side of most AI who do read through at least some of the chat data. We saw this with moderation on places like Xitter, and with Amazon's AI corner stores that allegedly charged things to your Amazon account allowing you to walk in, pick something up and walk out. That's just to name a few.

It's not that I don't also think that their claim that they don't train their AI on chat data is a complete farce. Just that I think it can be two things at once and one of those things they're more comfortable saying out loud. It's complete bullshit that they lie about either one, and honestly I think I should get to treat inanimate objects and services I pay for however I want within reason (I.e. I wouldn't be cool with anyone generating something illegal.like CP). I don't use AI but this whole "we'll ban you if you curse at our chatbot" thing is nonsense.