The thing about the IRS is that they exist mostly outside of the consideration of political pandering. They just need to enforce tax code, not care about what people think. Kind of like a computer program. They don't write the rules they just enforce whatever is on the books. They aren't really elected or responsible for the perception of their acts. They are already generally unpopular publicly so I don't see them being overly concerned about political shitstorms.