654
Motherboard sales are now collapsing amid unprecedented shortages fueled by AI (www.tomshardware.com)
submitted 3 months ago by to c/technology@lemmy.world
146 comments fedilink hide all child comments
 

cross-posted from: https://lemmy.bestiver.se/post/1091783

Comments

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 3 months ago (1 child)

You can use something like KoboldCPP on Linux, which allows both RAM and VRAM combined to run a model. O'course, not as fast when compared to pure VRAM or the Mac approach, but it is an option. I use my 128gb RAM with some GPUs for running models.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    Ollama and llama.cpp allow it too but it's super slow in my experience.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    Speed depends on how much of the model is on VRAM, and the dense/MoE architecture of that model. The RAM's benefit is more about having the ability to run the model in the first place. In any case, a dense Qwen3.6 27b would take up about 27-33gb-ish of memory, plus whatever context size you set.

    Upcoming implementation of MTP will increase the size of models, but in exchange, they will also run faster. About a 30%ish boost for dense models, a bit less for Mixture of Expert varieties, from the looks of it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    When I've tried running a ~14 gigabyte distillation of whatever model it is I was trying to run, it would come out super slow at I believe 50/50 GPU to CPU. It gets so slow it was just more bearable to run a 7 or 8 b model that would actually fit entirely in VRAM and run entirely on GPU. Also made the rest of computer usage more bearable.

    To be fair I do only have a 6 core 6 thread CPU though. It shot up to 600% usage so even the DDR4 memory wasn't really bottlenecking it. I suspect a 9950X would fare a lot better.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago

    I am using a 5950x, with 128gb of DDR4 3600 memory. The GPUs are a 3060 and 4090, totaling 36gb of VRAM. IMO, being bottlenecked by the CPU is definitely a thing, it just comes third after the VRAM and RAM considerations.

    With a 35b+3a MoE at Q8 with KV8, I get...

    [11:54:32] CtxLimit:18858/262144, Init:0.18s, Processed:17294 in 7.66s (2259.18T/s), Generated:1564/32768 in 29.01s (53.91T/s), Total:36.85s

  • permalink
  • fedilink
  • source
  • parent