Presumably, if they call, you're in some sort of personal relationship with the person, so whether or not you answer says something about how much you value that relationship. No? You trust that they only call if it's something important.

I only refuse calls from unknown numbers, never from someone in my contacts.