It's more an issue of consistency than of taste though with sweet potatoes. There are some fruits that also don't fit into a fruit salad for similar reasons, like e.g. passion fruit or very soft kinds of nectrarines.

But I have seen ginger in fruit salad before.