Every freedom ends where freedoms of others are infringed. That includes every freedom, let it be freedom of movement (you can go wherever, but not someone else's house), freedom of expression (you can express yourself however, unless that expression instills hatred towards others, inflicts trauma on kids etc. etc.) and yes, also freedom of speech (You can say anything, unless what you do is calling for violence, attacks someone etc.).

Some of you US guys really don't understand how freedom in a society works.