I would argue it wasn't just the refrigeration, but also the suburbanization of living and the cost effectiveness of delivering the milk from the farm to the store, which (in theory) made milk cheaper. You would still need the milkmen if stores and supermarkets didn't exist. In an alternative world where we didn't invent commercial / household refrigerators, you could still buy milk from stores daily without the need of a milkmen, becaue ultra-pasteurization exists.

I guess thats the problem with analogies and I don't think either of us will get anywhere by further arguing about this one specific example.