Free speech ends when it encroaches another's freedom. When you start saying things that could lead to someone actually being endangered or subdued in response, that's no longer protected. It can get more complex, but that's the starting point.

Free speech doesn't mean you can't be countered with someone else's opinion though. Some people seem to think they can say things and then shout their freedom is being taken away when someone else disagrees. There isn't harm there except to their opinion, and that is free to shoot down.

Another way to look at it is a rule commonly found on online forums. Attacking ideas is supported, attacking a person is not.

There’s nothing special about that phrase in particular (you just said it yourself).

The difference is between opinions (which are protected) and perceived threats (which are not).

Well, there was that one particular american president who got shot in the head. Then some century later, there was a second one. So I'd say there's two examples of precedent as to why it's something which evolved naturally, through the passage of time.