Or, if you take the machine learning approach, you just try all the combinations and use the one(s) that perform the best.

What I find LLMs doing for my software development is filling in the gaps. Thorough documented requirements coverage, unit test coverage, traceability, oh you want a step by step test procedure covering every requirement? No problem. Installer scripts and instructions. Especially the stuff we NEVER did back in the late 1980s/early 1990s LLMs are really good at all of that.

Nothing they produce seems 100% good to go on the first pass. It always benefits from / usually requires multiple refinements which are a combination of filling in missing specifications, clarifying specifications which have been misunderstood, and occasionally instructing it in precisely how something is expected to be done.

A year ago, I was frustrated by having to repeat these specific refinement instructions on every new phase of a project - the LLM coding systems have significantly improved since then, much better "MEMORY.md" and similar capturing the important things so they don't need to be repeated ALL THE TIME.

On the other hand, they still have their limits and in a larger recent project I have had to constantly redirect the agents to stop hardcoding every solution and make the solution data driven from a database.

Yes, this is why I point it out. I agree with you, but no part of this is actually common sense. It just feels like it.