If you think of it like everyone has their ideal "closeness" range, it makes more sense.
Some people like to be extremely close and become one combined person. Others like to keep partners at arm's length. Neither one is "wrong", they're just incompatible with each other.
If you get two compatible avoidant people, they tend to keep things casual and aloof, but know each other very well over the years. They just don't usually live in the same house.