I don't want to be measured as worthless by everyone around me

There's your problem right there. You need to learn to not fucking care one fucking bit what ANYONE else fucking thinks about how you and your family decide to live. They can all go FUCK themselves, and you don't have to worry about it even a little bit.

If your wife is cool with it, then create your own reality, rather than involuntarily play a role in someone else's.