Muscle mass is directly tied to how much you use the muscle.

I'd describe it more as a loose correlation rather than "directly tied." There's the obvious example of sex differences, where a typical untrained woman will not respond as quickly to weight training as a typical untrained man of similar size. There's a lot of individual variation, too, and more recent research is revealing new insights into high and low responders to resistance training. See, for example, this review.

I learned to be less of a dick to others about this stuff when I learned I was an outlier high responder in muscle strength and growth, getting much better results from near identical workouts with some of my friends. Realizing that I was lucky helped me extend some more grace to others.