Sure, sometimes you find requirements that you didn't think of beforehand.
But what is programming at the core? I'd summarize it like this: "Explaining how to solve a complex problem to a very fast idiot." And the thing C-Suits like to forget is that this explanation is given in a specialised language that, at least ideally, leaves no room for interpretation. Because ultimately the computer doesn't understand Python, Rust, C or even assembly. It understand opcodes given in binary. Assembly may be the closest human-readable approximation, but it still has to be translated for the computer to understand it.
So what happens when you "replace" programmers with neural networks? You replace a well-defined, precise language to use for your explanation (because you still have to explain to the fast idiot what you want it to do) with English or whatever natural language you train your network on. A language littered with ambiguities and vague syntax.
Were it a tool to drive nails into wood you would've replaced a nail gun with a particularly lumpy rock.

I don't see neural networks effectively replacing programmers any time soon.