I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.

It takes 3-15 minutes to run depending on how many tool calls the model makes and that's mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.

The kind of specs that would enable local LLM usage were available on laptops since 2020. The floor for acceptable performance for LLMs stopped literally just before 24+gb of VRAM would have become pretty common in consumer applications. The reason this technology isn’t more ubiquitously available is because ‘they’ thought it would be more profitable to hoover up literally every byte of RAM and GPU in the world rather than let the technology continue to naturally evolve to a place where any consumer could have run useful models on their home computers. In other words, your anger is misplaced.

Open models will still be available via API from third-party hosts. The bigger ones like Kimi K3 or GLM 5.3 don't fit on conventional desktop computers to begin with, most non-institutional users of those would already be doing that (eg via OpenRouter).

Anyone who lacks the skill to to it themselves…

More Ram, and break stuff up into smaller pieces. I Have 12Gb of VRAM and 64 Gig System Ram, which - with CPU offloading - allows pretty large models to be used at acceptable speed (although probably not fast enough for someone who wants to make his living with it. But these also will invest in dedicated hardware). And i let a larger model sit there to coordinate, and let it use subagents for the rest. That keeps the Context for the main model a lot more manageable and keeps it on track for a pretty long time.

How you structure the Task is more important than what the task is at all. For example, i have a huge list of games to be catalogued, and just telling the model to search the web and fill in the blanks in the database makes it crap out pretty soon. But I tell it to let subagents handle everything regarding web search and data extraction and just let them hand back the resulting data. the main model then calls another subagent to sanity check the data and update the database. Each of these subagents do not run for long, and therefor never run out of context, while the main model can keep context size manageable because it does not have to work with "messy" data at all.

I've got 24GB of VRAM, which helps a lot. I'm using the Q4_K_M version of this model, with the KV cache quantized to Q8, and that's able to get me to 128000 tokens of context. I'm using the Qwen Code harness in VS Code.

If 128000 tokens still isn't enough, break the task down into smaller chunks and help the harness out with MCP tools - I installed Codebase Memory to help it navigate large codebases more efficiently, for example. Ask the coding agent what's causing problems and it may be able to help identify solutions too, I've had it troubleshoot itself several times now and it's been quite good at inspecting its own settings and finding where the problems are with them. I even installed Firecrawl for it so it can do its own web searches.