I don’t know how the devs would figure out the error and how to fix it.

This is like the biggest factor that people don't get when thinking of these models in the context of software. "Oh it got it wrong, but the developers will fix it in an update". Nope, they can fix traditional software mistakes, LLM output and machine learning things... They can throw more training data at it (which sometimes just changes what it gets wrong) and hope for the best, they can do better job at curating the context window to give the model the best shot at outputting the right stuff (e.g. the guy who got Opus to generate a slow crappy buggy compiler had to traditionally write a filter to find and show only the 'relevent' compiler output back to the models), they can try to generate code to do what you want and have you review the code and correct issues. But debugging and fixing the model itself... that's just not a thing at all.

I was in a meeting where a sales executive was bragging about the 'AI sales agent' they were working, but admitting frustration with the developres and a bit confused why the software developers weren't making progress when those same developers always made decent progress before, and they should be able to do this even faster because they have AI tools to help them... It eternally seemed in a state that almost worked but not quite no matter what model or iteration they went to, no matter how much budget they allocated, when it came down to the specific facts and figures it would always screw up.

I cannot understand how long these executives wade in the LLM pool and still believes in capabilities beyond what anyone has experienced.