So walk away from them, or handle it gracefully somehow. It's the same as trying to make a friend. Doesn't everyone make friends from time to time?
Interest + effort = relationship of any kind. Find the shared interest, make a little bit of effort.... or don't and the math doesn't work. If romance doesn't come, you've made a friend.