I only cautiously use AI. Most certainly because I see how, AI companies apply "flex tape" to fix issues. They themselves don't realy know how the AI will behave and are only treating symptoms and ignore the real problem.

That's why I think your stance on "we don't 'really' know what AI is doing" is right. "Knowing the problem good enough to formalize the risks/policies" is a very valid point, but on the other hand, I think you would agree that we already solved such issues. Think about a "black box".

If you see AI as a black box, you might come to a similar conclusion as the author.

I know that this is not the fix we need or even want. But better this than playing the beta tester for AI companies.