This notion of “quality of output” misses something important though, which I think our economic system predisposes us to ignore.

I think most of us have this concept that the function signature of “working on code” is something like:

implement(spec): CodeChanges;

But I think this hides some stateful side-effects. There’s a monadic thing going on here, and I think it’s really more like:

implement(knowledge, spec): [CodeChanges, KnowledgeBundle];

…where KnowledgeBundle is a whole bunch of things: latent info about how things currently work, obviously, but also the yet-unnamed missing abstractions you noticed, the existing named abstractions that are starting to strain — basically the sense of the trajectory of the codebase.

Even really shitty programmers get some sense of the trajectory of the codebase, but this is something that even the best frontier models fail at.

Tangent:

I saw a pretty cool visualization of Kalman filters the other day (substantially AI-generated, coincidentally), and I thought it matched something about this dynamic.

• Whole video: https://youtube.com/watch?v=zFmO_b05jU4
• Short clip: https://youtube.com/shorts/vMRrWb_lf84

If you’re only concerned about the “next step”, aka “quality of output”, your error will accumulate and you’ll drift over time. You need to be able to evaluate where you are, so you can correct that error and stay on course. And to do that, you need to be present inside of the code and feel where the pain points are, or your estimation will be so far off that it’s useless.

I think our economic focus on “output” and our obliviousness to internal states (or even denial that they exist at all) is going to make it very easy to fall into the trap of taking a long series of next steps that individually look very sensible, but over time accumulate unmitigated error and drift towards disaster.