The rationale seems to be that it will learn all crooked if you talk to it nasty.
Which brings up two questions:
- Why is it learning anything from any interaction I have with it? Particularly when I have a paid-for agreement with the company that promises never to train their models on my code. Clearly that's a lie then.
- If the model gets all out of whack because I abused it, why don't they reset it? In fact, I fully expect Anthropic to reset the chatbot's sandbox each time a new customer uses it. Otherwise, by definition, the damn thing's performances might be affected by the previous use case, and it might barf out data I gave it to the next dude who uses it. This is basic data sanity. If they don't do that, they're fucking incompetent.