If we push too long and too far without maintaining or rebuilding coherence, with current trends, we'll end up in a place where we must use agents to maintain our systems, because the systems will be so large and so overwhelming that we cannot do so ourselves. But if we get caught there, and it turns out that agents cannot back us out by doing the repair work, we may find ourselves in a very uncomfortable position:

Locked into a development model that slowly degrades our systems no matter how much effort we put into fixing them.

It's not that we won't be able to eventually fix the systems. But it might take years or even decades to do so, and in the meantime, we won't be able to ship features or make progress on the many other pressing problems we have in our society.

This is what I call Decoherence Collapse. And I am far more afraid of it than runaway super-intelligence, because it's actually happening right now, in front of our very eyes.

Source [web-archive]