Diversity has repeatedly been shown to be more profitable than homogeneity, in both academic and gray literature. Besides being good for societal cohesion, fairness, stability, happiness, and moral virtue.

I did not say that diversity could not be more profitable, just that it is not always more profitable.

The best candidate is indeed best, but there are too narrow and outdated ideas on how to identify the best candidate, and humans have a bias to choose/hire for safety and similarity over actually relevant criteria, which is why we have the problem in the first place.

There is no way to solve this problem. I can be the best fit as person but the worst from a technical point of view, like you can be the best from a techincal point of view but the worst from a personal point of view.
Both of us would be a problem, although in different ways, in a team.