Understandable AI is an important field in machine learning, where it is about well understanding how the model came tobits conclusions based on the data. This is crucial to apply AI tools for anything beyond writing silly Haikus. An AI company that denies access to that basically wants its customers to use its tools like a fortune teller.

"Yes the computer read that in the stars. how why or how reliable the result? Dunno, but it says sobso it must be true. And now off to prison young black men, with a good job and no criminal record. The AI predicted you would commit a crime in 10 years."

EDIT: To give an example from a lecture i had. The task was picture classification and one model rekiabl, recognized pictures of a horse in the training data set, but failed to recognize it outside of it. Turns out all the pictures in the training set had a watermark text in the botton, that the model recognized as being the defining feature. And that is a very simple task in comparision.

Open AI wanting not disclose their training methods and data source indicates that there could be a lot of garbage like this in their models.