You're being too broad. Gotta specify what you're doing with coffee at that point, and a slight expansion.

Basically, when bringing it up make it sound more interesting and mysterious, with the potential of a follow-up story.

For instance, people often ask me what brought me to Los Angeles. I tell them, "it's lady Gaga's fault". Hooks 'em every time.

It's absolutely a hobby and a huge money pit.

^Askmehowiknow

Coffee brings up associations with hot swill at diners. Craft coffee or coffee nerd brings up a barista image, so maybe that's a good start.