There's no paradox if you look at it as a social contract. If you don't uphold your part of the contract (tolerating others) then you aren't entitled to benefits from the contract (being tolerated by others).