See, this is actually one of the rare times when violence is the answer.
One good kick to the nuts, and a "you ever fuck with me again, you'll sing soprano", and it's going to back the guy down. It doesn't have to even be super aggro, just casually roll up and do it, then say it calmly, and walk away.