I tend to read it as "they do not speak for me". If you don't speak up, then many people believe you agree with them, by default.

By apologising, you are saying you actively disagree, but don't want to get bogged down in the details. You might not stand directly with the victims, but you do stand beside them, against the abusers, and want them to know.