There's a wide range of "explainability" in machine learning - the most "explainable" model is a decision tree, which basically splits things into categories by looking at the data and making (training) an entropy-minimizing flowchart. Those are very easy for humans to follow, but they don't have the accuracy of, say, a Random Forest Classifier, which is exactly the same thing done 100 times with different subsets.

One flowchart is easy to look at and understand, 100 of them is 100 times harder. Neural nets are another 100 times harder, usually. The reasoning can be done by hand by humans (maybe) but there's no regulations forcing you to do it, so why would you?