654
Motherboard sales are now collapsing amid unprecedented shortages fueled by AI (www.tomshardware.com)
submitted 3 months ago by to c/technology@lemmy.world
146 comments fedilink hide all child comments
 

cross-posted from: https://lemmy.bestiver.se/post/1091783

Comments

you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 3 months ago (1 child)

That's interesting.

How much ram did it use while running?

If you used a GPU, how much does it cost in today's prices?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 12 points 3 months ago* (last edited 3 months ago) (3 children)

    It's a MacBook Pro. 36GB of ram. I am sure Macs have some kind of gpu and I understand it somehow combines GPU ram with system ram, but I don't really know Mac hardware very well.

    It's beefy for a laptop, but the desktop I built for myself several years ago had 32 GB of ram and a GTX 1660, so I'm guessing they are similar in capability. I gave that to my daughter, so I can't run a comparison right now.

    EDIT: After doing just a bit of research, I've learned the unified memory architecture that Macs use, while not ideal for many purposes, is actually a big advantage for running larger inference models. So it's possible that this particular model wouldn't run at all on my Linux box or would run much slower because the full model wouldn't fit in the 6GB of VRAM and create a lot of memory thrashing.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 4 points 3 months ago (1 child)

    You can use something like KoboldCPP on Linux, which allows both RAM and VRAM combined to run a model. O'course, not as fast when compared to pure VRAM or the Mac approach, but it is an option. I use my 128gb RAM with some GPUs for running models.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    Ollama and llama.cpp allow it too but it's super slow in my experience.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    Speed depends on how much of the model is on VRAM, and the dense/MoE architecture of that model. The RAM's benefit is more about having the ability to run the model in the first place. In any case, a dense Qwen3.6 27b would take up about 27-33gb-ish of memory, plus whatever context size you set.

    Upcoming implementation of MTP will increase the size of models, but in exchange, they will also run faster. About a 30%ish boost for dense models, a bit less for Mixture of Expert varieties, from the looks of it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    When I've tried running a ~14 gigabyte distillation of whatever model it is I was trying to run, it would come out super slow at I believe 50/50 GPU to CPU. It gets so slow it was just more bearable to run a 7 or 8 b model that would actually fit entirely in VRAM and run entirely on GPU. Also made the rest of computer usage more bearable.

    To be fair I do only have a 6 core 6 thread CPU though. It shot up to 600% usage so even the DDR4 memory wasn't really bottlenecking it. I suspect a 9950X would fare a lot better.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago

    I am using a 5950x, with 128gb of DDR4 3600 memory. The GPUs are a 3060 and 4090, totaling 36gb of VRAM. IMO, being bottlenecked by the CPU is definitely a thing, it just comes third after the VRAM and RAM considerations.

    With a 35b+3a MoE at Q8 with KV8, I get...

    [11:54:32] CtxLimit:18858/262144, Init:0.18s, Processed:17294 in 7.66s (2259.18T/s), Generated:1564/32768 in 29.01s (53.91T/s), Total:36.85s

  • permalink
  • fedilink
  • source
  • parent
  • [–] 3 points 3 months ago

    Yup, you want memory accessible to the GPU for local AI. AMD Strix Point and Mac devices are popular options. CPU can run LLMs but very slowly. I've got 32 GB of RAM and 8 VRAM and it's borderline useless for models that don't fit in the VRAM.

  • permalink
  • fedilink
  • source
  • parent