It's not only growing up with them. We're just better identifying people/animals/things we're familiar with. Horses all look the same if you're not around them regularly. You can distinguish colours, but that's it.
Not comparing people to horses by the way...