Random

It doesn't need to be, but English speakers have a habit of arbitrarily choosing the gendered word that they import from languages with gendered adjectives and nouns. For example, even men massage therapists are "masseuses" and not "masseurs".

Somewhat strangely, we have slowly dropped our own gendered titles over the decades. We no longer say "manageress" nor "doctoress" and many people have stopped saying "actress".

English speakers as a large group seem to not try to maintain any logic to their choices. They merely copy others and don't think much about what they're saying. If you refuse to use "petite" like that, it only increases the chances that others stop, but we would probably need some influential person to criticize it---but not too loudly!---for larger segments of the English speaking world to change their habits.