I understand where you're coming from, and your boundaries should be respected. Which is why I did not make an issue out of it. Like I said, we're still very good friends. And when he waved me off I just laughed awkwardly back and went with it.
And...I know him really well. It isn't that he doesn't like being touched. He, like a lot of men in America, is super phobic about anything within 500 miles of "gay".