So, I was thinking about how modern LLMs got better at coding in the last 1 year, despite not having much good new training data.
The way it was done is largely via reinforcement learning with verifiable reward (and a little bit with human feedback).
Basically, you use an LLM to make slop, random token choice and everything, monkeys on typewriters style, and then if slop passes a test, you use said slop as the training data. You use the compiler, unit and integration tests, any static analysis tools, etc etc as verification.
For natural language, attempts to do anything like this work miserably, the whole slop centipede, model collapse, all that. But for coding this turned to be unexpectedly effective.
So that raises the question: can an LLM coding assistant be developed like a normal corporate project, with no YOLO exemptions from software legal compliance, without any special treatment like it's a favorite child that grows up to be a techbro? Like normal fucking software?
I think, the answer is probably a yes on the purely technical level but still a hard no commercially.
On the technical level, an LLM could be pre-trained on compatibly licensed open source projects, RLVR'd in a license-compliant way.
The end result isn't going to be good for plagiarism, though, and won't use plagiarism to cover every gap in capabilities. I think it's pretty obvious all the folks who are trying to clone Adobe or whatever would reject such a product.
Meanwhile the one "strength" of LLMs is this intuitive argument that if you have a good enough predictor of what your workers do, then you don't need workers. That is the whole reason the "AI companies" are rolling in investor money and are able to operate at such large loss.
The other thing is that having done copyright infringement, OpenAI/Anthropic/etc etc have to portray it as necessary, and alternative as impossible.
Anyhow basically my thought on this is that I'll continue not using any LLM coding tools in my personal projects until some acceptable tooling appears and proves itself actually useful.
edit: For a concrete example if someone makes a tool that can take the compiled assembly, run it through non-LLM translation to some much abridged Lean and prove that nothing depends on exact timing of threads (no race conditions) or identify race conditions, or complain it could do neither, would I use it? If the carbon footprint is not insanely greater than what I would do instead (which is either a: run stress tests overnight, or b: ship buggy software that users have to restart, which also has a large energy footprint), sure.
