I suppose in a way you are right, as I can't really speak for anyone else's concern, but his comments are complete speculation that attempt to attack the character and not the argument of the conversation, also derailing it. There's no way for him to prove his claims, and it just acts to imply that someone could have been banned so their thoughts don't matter.
Imagine someone banned my account for these comments, would they cease to be relevant? Would they cease to be relevant if I brought my thoughts up on an alt? I guess you could question the motive, specially if the conversation was toxic, but have my questions and concerns been toxic? So even if it became applicable, would his doubts be relevant?