This is what I used to do, the smoking area outside was a nice little shaded area, i just generally stood upwind of the smokers or slightly askew to avoid the 2nd hand smoke. No one gave me shit, I was just asked occasionally if i smoked.

One of my colleagues will even occasionally ask me "Heading for a smoke, wanna come along?" I just love chatting with him, I'll try to stand upwind so I don't catch as much second-hand smoke, he gets some company too, everyone's happy.