That statement works for every other freedom you lose, it also serves to detect malicious intent from another person. There's always a middle ground, where nothing's perfect, but it's balanced. There's always a compromise. There's no perfect scenario. If you want a perfect society, you have to take away all freedom. If you give away all freedoms, there's anarchy.