My guess is the next big thing to come out is, we can probably squeeze a lot more reliability out of smaller models. But their workflows, context, validations, etc will need to be very tightly optimized.

I can see harnesses coming with their own highly specialized lightweight models in the future. Some for very efficiently converting a basic prompt into chain-of-thought steps. Some for very efficiently determining relevant parts of a repository. Some for… a lot of highly specialized stuff. Then the harness would orchestrate these under the hood, reducing the cognitive load placed on any larger generalized LLMs. Those “larger generalized LLMs” could be something like 12b parameters.

Hopefully, soon after, we can start benchmarking how much different harnesses and augmentations improve baseline model performance. Ideally, in the long run, with a deeper understanding of how to tailor harness to workload and produce more procedural determinism. Then we can start configuring harnesses like data pipelines and run them through higher-level orchestration like Airflow too.

Yeah LLM's can help with many tasks but then there are times they just spout nonsense, or syntactically correct nonsense, the model size and context window just changes when they hit their limit.

Sometimes you have to call it quits, and try another way.

Most developers use their work provided machines, which aren’t gaming machines with giant GPUs because again, GPUs don’t help development at all.

Also developers often want more ram, and if youre on the mac side, the M series ram works as video ram for loading and running models, so there's a good chance you can already run something better than is typical of others, and apple is focusing on this by adding more NPUs and increasing memory bandwidth. They arent good at training, but can do inference.