I am more likely to lose my wallet than my phone. Replace 'phone' with 'wallet' in your comment and your whole point is lost. Yes, it would be rather inconvenient to lose my phone. Far more inconvenient than losing my wallet.

At least use spying/tracking my movements as an argument against the phone. While also being a bit paranoid, it is at least a scenario that is possible. And I would actually change my habits of I thought it likely.

Edit: In an ironic twist, I realised about three-quarters of the way to work this morning that I'd left my bag (including wallet) at home. I have my phone and keys, so I'll be ok. Not having my security pass is the biggest hassle.