From TFA:
The AI did not prove that its approach is the best anyone can do, though. In fact, mathematician Will Sawin has already improved upon the AI’s grid.
OpenAI privately contacted Litt, Sawin, Gowers and a number of other mathematicians to verify the LLM’s proof. Together (and without the company’s direct involvement), they wrote up their individual takeaways. (No external experts have seen the AI’s original output, however—just an edited version of its train of thought.)
What stood out, they said, was the AI’s preternatural patience and focus.
...
“AIs have an edge: It’s not just that they can try all known methods,” says Jacob Tsimerman, a mathematician at the University of Toronto, who was not involved in the work but was part of the companion paper solicited by OpenAI. “They can play for longer and in more treacherous waters than mathematicians without getting overwhelmed.”
...
The mathematical tools the AI used here are not novel, although their application in this domain appears to be. “The model did not invent something fundamentally new that nobody saw coming,” says Sébastien Bubeck, a mathematician leading OpenAI’s mathematical explorations. “It just executed like an amazing mathematician.”
So, it's a monkeys-on-typewriters situation with the computer able to try and reject the hammering of who knows how many square pegs into round holes until it finally arrives at a workable conclusion, which a human has already bested. And we're not allowed to see its homework.
This is categorically failing to set the world on fire, except possibly in the literal sense.