This one gets to me. Not because I liked them that much, but because Justin has always been a dick and I could just write it off as "yeah they were a bunch of dicks but they put on a good show". An Anti-Flag show in the 2000s, here at least, was one of the few times a punk show wasn't in a basement somewhere, so it was absolutely THE place to be if you were in the scene. They also frequently had great bands with them so it always made for a great time.
But after the accusations came out it left a real bad taste realizing that he's probably much more than "just a dick".
As someone also in their mid 30s, I think a lot of people really misunderstand how the confederate flag was viewed so casually by many people up until recently. It was always offensive to some, but I genuinely was unaware of that as a teen growing up in the 2000s. It was just an image of being kind of edgy and rebellious but I genuinely didn't think much of it seeing any band using them. Watching Skynyrd or Guns N Roses videos of live concerts? There were "rebel flags" pretty often and I just kind of let it go. It meant nothing to me.
I was in my mid 20s, probably, when it really was explained to me how insensitive that can be. I never associated it with racism or slavery up until that point - and I'm pretty sure that's the same thought that a lot of these bands had. The absolute best thing to do is exactly as you said: own up to it, change your ways, and grow.