973
Managers (thelemmy.club)
submitted 4 months ago* by to c/whitepeopletwitter@sh.itjust.works
180 comments fedilink hide all child comments
 
Managers
you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 4 months ago* (6 children)

At Q8 it is around 35-40GB I think + memory for required context.

I have a Framework desktop. It gets you you around 6t/s. Not suitable for professional use but for personal use I think it is fine. I do prefer Gemma 4 though, but that comes with similar reqirements.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 1 point 4 months ago (5 children)

    huh, i thought that ryzen ai thing would perform better than that. my 7900xtx regularly gets 30+tps with qwen, up to hundreds with more compressed models.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 4 months ago* (4 children)

    My system runs at 100W TDP though. That is maybe 140W at the power outlet, incl. monitor and everything.

    This is also the dense 27B model at Q8. But yeah, it is not terribly fast. I think the best use case is on MoE models. GPT-OSS-120B runs on it for example and at 50T/s speed is not a n issue anymore either. (I could get it to run even on just 64GB but the new llama.cpp might need a tiny bit more memory which pushed it just across the limit. yeah I know, for seriously using it you'd need the 128GB version)

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 4 months ago (3 children)

    that's fair, i'm at like 7x the power. the gpu alone easily pulls 350-400W and the rest of the system isn't exactly running lean either.

    ...man now i really want more vram.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 4 months ago* (last edited 4 months ago) (2 children)

    Yes I think Strix Halo makes sense when low power use is a requirement. I built a custom fanless Strix Halo system for the fun of it and I guess there aren't too many out there running Gemma 4 31B Q8 without a single fan, anywhere.

    And for MoE models that need 60-80GB + context it is perfect. Those are decently fast then as well.

    PS: If VRAM is all you care about the maxed out Mac Studio is fascinating. 512GB unified memory for around 10K EUR (pre crazy bubble prices) That should be able to run pretty large MoE models but dense models of that size would probably run glacially.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 months ago (1 child)

    i'm not buying any hardware for the forseeable future :P it's all just wishful thinking at the moment. but unified memory architectureis probably going to become more common so maybe in five years when some new motherboard standard becomes the norm...

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •