We have computing power in our pockets a million times more powerful than we used to send man to the moon, why do you think we'll never have enough power?

Not the person you replied to, but I have thoughts on this point in particular:

1. Consumer devices have started to slow down their performance improvements because we're bumping up against the limits of physics
2. People/corporations with way more money than the average consumer will always be able to run something orders of magnitude more powerful. Any advances that improve things for the average consumer will improve things for rich people/corporations even more.
3. Training an LLM isn't really even about compute speed, it's about access to good training material. The average consumer can't afford to buy (or pirate) every book in existence like a rich person/corporation can. An average person doesn't have the ability or time to curate their own training data, but rich people/corporations do.

22B

There are 700B+ parameter open weight models now. Frontier models are in the trillions.

And even that model apparently took a supercomputer to train. I don't have a supercomputer so I can't train my own models like I can compile my own software. This is not comparable to running Linux where you can just compile your own kernel or even whole operating system (former Gentoo user here).

I've tried running the models my 8 GB card can handle. They're OK for a quick question, but they won't be doing anything useful for me.

Because companies are using so much computing power it requires as much electricity as a city. Or you can take your pocket computing resources and see his long it takes to train an LLM.

These “flagship” models with hundreds of billions of parameters and giant context windows have not achieved proportional gains in benchmarks.

It's not proportional, but they're still far more useful. And the giant context window is great when working with a nasty complex codebase (hey, I didn't write the bloody thing, I do modules for an open source ERP). The entire codebase would be tens or hundreds of millions of tokens, but any useful portion of it is still tens or hundreds of thousands.

You know what does significantly improve accuracy, even on smaller-sized LLMs? Retrieval-Augmented Generation. There’s literally no reason to use giant, resource-intensive models. You can just give a smaller one access to databases and libraries with all the information it needs to report on.

Larger LLMs still have RAG.

A 24 billion parameter model is easily self-hostable on consumer hardware, and can be quantized to further reduce hardware requirements for a marginal loss in accuracy. At 8-bit quantization, that only requires 24GB of RAM + overhead. A lot by some standards, but by no means unachievable for a hobbyist.

And a 24 GB graphics card is like a thousand euros. I could afford it, especially since I can claim VAT back, but the majority of the planet's population can not unfortunately. You could run it on system RAM, but when I run partially GPU/VRAM, partially CPU/RAM, it gets pretty slow for me. And I'm not even talking about a 24 GB model, I'm talking about qwen3:8gb because I unfortunately only have an 8 GB card right now. I built this PC for gaming not LLMs and it's a few years old.

Maybe once I upgrade to DDR5, it'll be bearable on CPU though.

Most of the models on huggingface are pre-trained and list their datasets. You can fine-tune and align them yourself, which uses far less resources than pre-training does. You can even use LoRA to further reduce resource needs.

But that's what I'm talking about, we can only really run pre-trained models locally in the foreseeable future. Pre-trained by who? Usually some AI company (DeepSeek, Qwen, etc). Training is hard even without the hardware requirements, you also need to hoover up tons of data. So the current, already existing, models we'll be able to run, and thanks to RAG they won't ever really be fully out of date like the early iterations of ChatGPT were, but they won't compete all that well compared to whatever models we'll have in a few years. And even then we're relying on commercial search engines that the LLMs can use to get access to new data.

Don’t fall for the lie that commercial LLM APIs are the only viable option.

They may not be, but 20€ a month for an LLM that responds quite fast is more affordable than shelling out tons for a better graphics card to get a slow running and less capable LLM. Okay, sure, that's opex vs capex, but the graphics card also takes a bunch of electricity to run so it kinda evens out on the opex as well.

Commercial AI is currently the only thing that can help me do a day's worth of refactoring in 30 minutes. Yes I'll still have to check the output critically. But overall it can handle such things, as long as there is good test coverage and you understand what the code is supposed to do. I've tried using a local LLM for this and it took several minutes to essentially tell me it has no idea what I want it to do. That I believe was qwen2.5-coder:7b and the default context window, though I've since upgraded to qwen3:8b and increased context window from the default 4096 that ollama thinks is appropriate for my system (causing the aforementioned offload to CPU/system RAM, making it even slower) to... well 64k in the config, but it's only really showing up as 40960 in ollama ps so I'm guessing I hit the limit for either the model or my hardware.

If you can show me a local LLM working properly on consumer-grade hardware and doing useful coding tasks in a reasonable amount of time, I may just upgrade my hardware to run a local LLM, but until then I'm thinking of getting an additional subscription to complement Claude Code when running out of tokens. Probably Mistral because fuck ClosedAI. I did Google people's experiences with running Claude Code and the like on local LLMs and it appears even people with good hardware and better models are having subpar experiences

For just asking questions though, you're right, a local LLM works great. It's just not what I need out of an LLM much of the time.