639
Framework’s first desktop is a strange—but unique—mini ITX gaming PC (arstechnica.com)
submitted 2 years ago by to c/technology@lemmy.world
282 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 2 years ago* (5 children)

Fair. True.

If your workload/test fits in 24GB, that's already a "solved" problem. If it fits in 48GB, it's possibly solved with your institution's workstation or whatever.

But if it takes 80GB, as many projects seem to require these days since the A100 is such a common baseline, you are likely using very expensive cloud GPU time. I really love the idea of being able to tinker with a "full" 80GB+ workload (even having to deal with ROCM) without having to pay per hour.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 2 years ago (2 children)

    This is my use case exactly.

    I do a lot of analysis locally, this is more than enough for my experiments and research. 64 to 96gb VRAM is exactly the window I need. There are analyses I've had to let run for 2 or 3 days and dealing with that on the cloud is annoying.

    Plus this will replace GH Copilot for me. It'll run voice models. I have diffusion model experiments I plan to run but are totally inaccessible locally to me (not just image models). I've got workloads that take 2 or 3 days at 100% CPU/GPU that are annoying to run in the cloud.

    This basically frees me from paying for any cloud stuff in my personal life for the foreseeable future. I'm trying to localize as much as I can.

    I've got tons of ideas I'm free to try out risk free on this machine, and it's the most affordable "entry level" solution I've seen.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago* (1 child)

    And even better, "testing" it. Maybe I'm sloppy, but I have failed runs, errors, hacks, hours of "tinkering," optimizing, or just trying to get something to launch that feels like an utter waste of an A100 mostly sitting idle... Hence I often don't do it at all.

    One thing you should keep in mind is that the compute power of this thing is not like an A/H100, especially if you get a big slowdown with rocm, so what could take you 2-3 days could take over a week. It'd be nice if framework sold a cheap MI300A, but... shrug.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 2 years ago

    I don't mind that it's slower, I would rather wait than waste time on machines measured in multiple dollars per hour.

    I've never locked up an A100 that long, I've used them for full work days and was glad I wasn't paying directly.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 2 years ago (1 child)

    Yeah, I agree that it does help for some approaches that do require a lot of VRAM. If you're not on a tight schedule, this type of thing might be good enough to just get a model running.

    I don't personally do anything that large; even the diffusion methods I've developed were able to fit on a 24GB card, but I know with the hype in multimodal stuff, VRAM needs can be pretty high.

    I suspect this machine will be popular with hobbyists for running really large open weight LLMs.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 2 years ago*

    I suspect this machine will be popular with hobbyists for running really large open weight LLMs.

    Yeah.

    It will probably spur a lot of development! I've seen a lot of bs=1 speedup "hacks" shelved because GPUs are fast enough, and memory efficiency is the real bottleneck. But suddenly all these devs are going to have a 48GB-96GB pool that's significantly slower than a 3090. And multimodal becomes much more viable.

    Not to speak of better ROCM compatibility. AMD should have done this ages ago...

  • permalink
  • fedilink
  • source
  • parent