LLM vendors are starting to charge money. I’m sure it’s not even close to profitable but it’s a start. Perhaps when the bubble pops and market consolidates, fewer vendors with more paying customers each …

Using an LLM is a skill just like any other. If you just take what it gives you, you can’t expect good results. If you evaluate what it gives you and prompt it to improve, the results aren’t as bad.

I use an LLM for coding and a definitely a skeptic, but I do find it a useful tool and am really interested in seeing if I can make it work.

Initially I found some amount of success at lower levels, saving me some time

• it could auto complete entire lines of code (and that’s trivial to evaluate and correct if necessary)
• it was pretty good about generating unit tests since they tend to be simple and repetitive. In general corrections tend to be smarter coverage, tweaking the tests to cover more functionality with fewer tests
• it’s pretty good with utility scripts. For example today I had a decision and wanted supporting data: in minutes it generated a script to call APIs in my scm and generate some stats for 4,000 code repos …. And it worked

Currently I’ve created rulesets and project context so

• it’s been quite successful at code reviews (it finds things I miss, and has resulted in my human reviewers finding less)
• I’m proud of one for identifying refactoring opportunities. It finds good spots and makes good suggestions, but so far I have to implement myself: its code hasn’t been usable. I can also objectively verify by reduced cyclomatic complexity.

Trying to find other scenarios it can be successful, it’s clear that insufficient context is a limiting factor. The fun challenge is to see if there are more successful scenarios if you can give it enough context. I’ve gone past rulesets and project context, to connect relevant services and metadata about our product set and environment. They want a team to try vibe coding and I’m still very skeptical, but my part of the effort is a real solvable problem and fun challenge whether they succeed or not