I won't claim this is nearly the same magnitude of issue but this comment reads like a "what was she wearing" type of comment. You're presuming a particular perpetrator and victim and immediately doubting a portrayal and envisioning your own because they don't agree.
I'll just say that, many of the times I have opened up to some people, I have come to regret doing so. And no, the stress was not "I said something racist and everyone hates me now". I won't detail beyond that for obvious reasons, but I know where the sentiment comes from.