This case of AI models solving mathematical problems is a happy little accident falling off a wagon of problems. This new technology, if it were developed to solve mathematical problems, wouldn't do nearly as much harm. Alas, the only mathematical problems OpenAI care about are from the realm of Game Theory, applied to corporate decision-making and boil down to "how do I make the most profit relative to the other players?"

It's like bombing a city to show off your cool bombs, finding a hostile in the rubble and pretending he was your target, as if that would justify all the other people you killed. And now you're going on about the way bombs can be used to kill bad guys in the context of a bomb manufacturer that doesn't give the faintest fuck about the bad guys, nor about the children they also killed, so long as the bombs are being bought.

Yes, of course AI can have good sides. Are you really surprised that we're more concerned about the bad ones right now though?