I find that the LLM agents find maybe 90% of "bugs" in their first pass, and 2/3 of those tend to be stuff I really wouldn't sweat over in most cases, but O.K. they're easy enough to button up so let's do that, then the other 30% are more serious stuff I'm genuinely relieved got found sooner than later.

Then you get into that last 10%, and I'll help it in the hunt, and probably 90% of them I'll solve much more quickly with the LLM agent than I ever would have without it, even if I have to give it directions about where and how to look for the problems / solutions.

Then there's the final 1% where the LLM just doesn't figure it out, even with clues, and I can see it and fix it far faster than the LLM... those exist, and they make great "how dumb are LLMs these days???" stories, but that's ignoring the other 99% where LLMs are clearly helpful.

Of course all the above is generalization, and if you roll back 12 months ago the LLMs weren't nearly as good then - and any further than that and I thought they were a total waste of time, but starting about 12 months ago it was pretty clear they were useful, and improving, fast.

Last week I actually solved a network configuration problem on the other side of the planet by having the LLM: 1) create scripts to diagnose the problem, 2) have a test engineer run those scripts on problem machines and control (non problem) machines, 3) get the logs from those script-runs and have the LLM analyze them, 4) repeat the process, based on information from 3 make new scripts focused on the problem area, 5) more test engineer runs, this time generating many megabytes of log files, 6) LLM analyzes the log files from 5 and identifies the issue, 7) LLM makes a script to reconfigure the terminal machine to solve the issue, 8) test engineer deploys it and retests - problem solved, 9) I discover the "solution" broke something else, describe the problem to the LLM, 10) LLM adjusts the "solution" to fix the problem without creating the new problem.

What's more remarkable about all that is: we first observed this problem 5 years ago, it was kind of a minor annoyance how it manifested but it's recently become more important that we solve it. In those 5 years, nobody here or there figured out what was going wrong, but a couple of rounds of applied LLM diagnosis and treatment led to a solution inside a week.