I’m really looking forward to the day when investors finally realize that spicy autocomplete isn’t particularly interesting, and is decades away from being able to do anything like what OpenAI is claiming (if it ever gets there at all).
I’ve been working with these models for years, even before ChatGPT came out. I’ve got a whole slew of private benchmarks I run these models through that haven’t leaked into the training data because I’ve never published them. I’ve tested all of them aside from ChatGPT-4.5 because it’s insanely expensive, but even the best ones are fucking terrible at logic.
They’re useful for problems that have been posted to StackOverflow a bunch of times. They’re borderline useless for anything truly novel. Pick a hard problem that isn’t on the Internet and give o3 or Claude Sonnet 3.7 a try. It’s hilarious how badly they perform. The problem doesn’t even need to be that hard. I’ve got logic problems I solved in under a minute that even the best models completely fail to solve.
But I don’t doubt that fraudulent claims like the ones being made by OpenAI will lead to significant job losses, and probably people losing their careers and even committing suicide. The sheer level of evil behind all of this is just staggering.
In a just world, the executives of these companies would go to prison for the damage they’re causing with their lies, economic malfeasance, and outright fraud. It’s truly staggering how amoral these people are.