40
Hardware for local inference? (lemmy.ca)
submitted 4 months ago by to c/selfhosted@lemmy.world
27 comments fedilink hide all child comments
 

I want to host some LLM's locally and use more advanced models. Since new hardware is out of the question, I think I should be able to pull something off buying some yesteryear equipment on ebay etc. Did anybody attempt such a project? Does it scale horizontally? (I.e. can I connext two boxes to overcome single box slowness?)

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 2 points 4 months ago (2 children)

well, I intend on scavenging for parts as I can't really afford today's prices. And since I don't really know what should I grab as minimum specs I don't even know what to look for. I could try to look for old(er) gaming rigs people sell or maybe there are some business workstations that may be sold in bulk. Either way, knowing what's the minimum viable set of specs for running qwen or claude locally would be helpful

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 4 months ago

    Word of advice - don't scavenge old /server/ hardware if you plan to put GPUs in - unless you really like heat and noise. Those machines from the likes of HP will take one look at a consumer graphics card in a PCI slot and decide they need to run all the fans at 100% to ward off evil spirits. Not to mention just the general ballache of proprietary PCI riser cards, PCI power cables, etc. etc.

    You're definitely better off with taking someone's old gaming rig off their hands.

    In terms of specs - value VRAM above everything else. A slow, old 3000 series card with 24GB of VRAM is much more useful than a brand new 5000 with 16GB. If you can find old RTX3090 24GB, they're kinda ideal.

    The one thing I will say for modern cards though is that they're much better for power efficiency - and in particular idle power (which is important if you're running the thing always on.) For my main LLM machine I have two RTX5060Ti (32GB total), which at the time was the sweet spot for price/performance/power, and it's very nice that they idle around 3 or 4 watts. I bought them before the world went crazy and prices went mad though, so they may not be the sweet spot any more.

    One you're in 32GB VRAM type territory, you can run really really good dense models like Qwen-3.6-27b at a decent quant, decent context size, and good performance for things like coding, or bigger MoE models for more general use (particularly then if you have good CPU and regular RAM for offloading to CPU. For use as an assistant (i.e. not an OpenClaw fully automated slop machine,) I use 3.6-27b as a daily driver in Claude Code, and basically never use Sonnet.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 4 months ago

    It can vary a lot based on what qwen model you want to run, but generally the 27b dense or 35b MoE are currently the best balance between size and capability afaik.

    If you can run two 16GB cards you can pretty much max out the context on the 27b model, but a single card like the 3060 12gb could still work well on the 35b MoE model with the excess spilling into system memory.

    I saw in another comment you have cards from the 2010's but if they don't have at least 8gb I wouldn't even bother

  • permalink
  • fedilink
  • source
  • parent