That's the problem when they never or rarely properly identify themselves. You can never truly know if they are to be believed.

They more than likely were with ICE. The agency has massively and rapidly expanded its enforcement personnel, and has done so in a haphazard way that has (deliberately) not given them an opportunity to properly train personnel up on procedures and rules and stuff. Nobody who wants to do good in the world is going to work for ICE right now. So, they've basically just done a presto gestapo, and a lot of times these guys really are just out there doing the sloppiest, shittiest, and most bully-like law enforcement you could possibly conceive of. I know where I'm posting this, I know what we all generally think of cops here, the current state of ice is about ten rungs down from that. We're talking real amateur hour shit, and they'll get away with it if people let them.

The good news is that apparently none of these fucking jokers seem to know or care how to cross t's and dot i's. You absolutely can send them packing, just like these folks did, and if they don't leave, then call your local sheriff's office or police dept. This is really dependent on your area, metro PD tend to be stocked full of major league assholes, but my local PD is actually pretty chill and has a good working relationship with the community; in my experience, sheriff's deputies tend to be people who got into law enforcement because they're true believers in the ideals of law enforcement (protecting life and property, accountability, etc) and constitutional justice, not because they want to flex on minorities. YMMV.

But if you think you can trust one of your local law enforcement agencies, absolutely call up 911 and tell them that you've got some people here that are trespassing (they've been told to leave and aren't), seem to be impersonating law enforcement (since they won't provide proof of ID), and you're worried that they're trying to kidnap someone, that you think these people are an imminent threat to your life or someone else's. That'll generally get the cops out there in a hurry, and they will likely be not very happy at all that ICE is out here putting on a fucking circus. That, or the agents will hear you and fucking split so that they don't get clowned on by local law enforcement. There have been plenty of instances of federal agents getting arrested while fucking around; mind you, they're almost never charged, but nobody walks away from that particularly happy.

there is no such thing as a “legitimate ice agent” ;)

I doubt they were actually ICE either.