The absurdity of their lack of consistent uniform and identification aside, in this hypothetical the people start impersonating not simply because of their tacticool cosplay like ICE, but by being among them and accusing others of being the fakes, implicitly asserting they are the genuine ones. So if you happen to be tacticool cosplaying in the vicinity of ICE agents without taking action, then maybe you are in a defensible. But acting on that adds an element.

Of course, this would mean something except when dealing with a 'shoot first, ask questions never' scenario...