These are baseless assertions - Your skin is covered in a layer of oils and skin that will capture your grime, and will naturally come off with hands and soap in running warm water. Unless you're regularly covered in like glue or you're showering every hour then using your hands will absolutely work to the same efficacy.

Exfoliation happens naturally, it does not need artificially scrubbing and materials unless you have a medical condition. My own baseless assertion is that scrubbing with a sponge or rag will remove more oils or skin than you should and instead expose and dry out your skin, making it worse.