This reminds me of another post I'd read, "Hey, wait – is employee performance really Gaussian distributed??".

There's this phenomenon when you're an interviewer at a decently-funded start-up where you take a ton of interviews and say "OMG developers are so bad". But you've mistakenly defined "developer" as "person who applies for a developer job". GPT3.5 is certainly better at solving interview questions than 90% of the people who apply. But it's worse than the people who actually pass the interview. (In part because the interview is more than just implementing a standard interview problem.)