Despite the “official” coding score for GPT5 being higher, Claude sonnet still seems to blow it out of the water. That seems to suggest they are training to the test and the test must not be a very good test. Or they are lying.

In a similar situation. I'm even an AI proponent. I think it's a great tool when used properly. I've had great success solving basically trivial problems with small scripts. And code review is helpful. Code complete is helpful. It makes me faster, but you have to know when and how to leverage it.

Even on tasks it isn't good at, it often helps me frame my own thoughts. It can identify issues better than it can fix them. So if I say here is the current architecture, what is the best way to implement and explain why, it will give a plan. It may not be a great plan, but as it explains it, I can easily identify the stuff it has wrong. Sometimes it's close to a workable plan. Other times it's not. Other times it will confidently lead you down a rabbit hole. That's the real time waster.

"Why won't the context load for this unit test?"

You're missing this annotation.

"Yeah that didn't do it. What else."

You need this plugin.

"Yeah it's already there."

You need this other annotation.

"Okay that got a different error message."

You need another annotation

"That didn't work either. You don't actually know what the problem is do you?"

Sad computer beeps.

To just take the output and run with it is inviting disaster. It'll bite you every time and the harder the code the worse it performs.

Dude forgetting stuff has to be one the most frustrating parts of the entire process . Like forgetting a column in a database or just an entire piece of a function you just pasted in... Or trying to change things you never asked it to touch. So freaking annoying. I had standing instructions in it's memory to not leave out pieces or modify things I didn't ask for and will put that stuff in the prompt and it just does not care lol.

I've used it a lot for coding because I'm not a real programmer (more a code hacker) and need to get things done for a website, but I know just enough to know it's really stupid sometimes lol.

Out of curiosity, do you feel that you would have been able to write that new function without an LLM in less time than you spent fighting GPT5?

LLMs are decent with coding tasks if you know what you’re doing

Only if the thing you are trying to do is commonly used and well documented, but in that case you could just read the documentation instead and learn a thing yourself, right?

The other day I tried to get some instructions on how to do something specific in a rather obscure and rather opaquely documented cli tool that I need for work. I couldn't quite make sense of the documentation, and I found the program's behavior a bit erratic, so that's why I turned to AI. It cheerfully and confidently told me (I'm paraphrasing): oh to do "this specific thing" you have to use the --something-specific switch, and then it gave some command line examples using that switch that looked like they made complete sense.

So I thought: oh, did I overlook that switch? Could it be that easy? So I looked in the documentation and sure enough... the AI had been bullshitting me and that switch didn't exist.

Then there was the time when I asked it to generate an ARM template (again, poorly documented bullshit) to create some service in Azure with some specific parameters. It gave me something that looked like an ARM template, but sure as hell wasn't a valid one. This one wasn't completely useless though, at least I was able to cross reference with an existing template and with some trial-and-error, I was able to copy over some of the elements that I needed.

More and more, for tasks like that I simply will not use an LLM at all. I'll use a nice, predictable, deterministic script. Weirdly, LLMs are pretty decent at writing those.

have to agree on that, there's the variation, it's faster if you take it's code verbatim, run it, and debug where there's obvious problems... but then you are vulnerable to unobvious problems, when a hacky way of doing it is weak to certain edge cases... and no real way to do it.

Reading it's code, understanding it, finding the problems from the core, sounds as time consuming as writing the code.

On the code competition, I think it can do like 2 or 3 lines in particular scenarios. You have to have an instinct for "are the next three lines so blatantly obvious it is actually worth reading the suggestion, or just ignore it because I know it's going to screw up without even looking".

Very very very rarely do I find prompt driven coding to be useful, like very boilerplate but also very tedious. Like "show user to specify these three parametets in this cli utility", and poof, you got a reasonable argv handling pretty reliably.

Rule of thumb is if a viable answer could be expected during an interview by a random junior code applicant, it's worth giving the llm a shot. If it's something that a junior developer could get right after learning on the job a bit, then forget it, the LLM will be useless.

Not yet. I'll give them a shot if they promise never to say "you're absolutely correct" or give me un-requested summaries about how awesome they are in the middle of an unfinished task.

Actually, I have to give GPT 5 credit on one thing: It's actually sort of paying attention to the copilot-instructions.md file, because I put this snippet in it: "You don't celebrate half-finished features, and your summaries of what you've accomplished are not only rare, they're never more than five sentences long. You just get straight to the point." And - surprise, surprise - it has strictly followed that instruction.

Fucks up everything else, though.