639
Framework’s first desktop is a strange—but unique—mini ITX gaming PC (arstechnica.com)
submitted 2 years ago by to c/technology@lemmy.world
282 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 72 points 2 years ago (15 children)

Not really sure who this is for. With soldered RAM is less upgradeable than a regular PC.

AI nerds maybe? Sure got a lot of RAM in there potentially attached to a GPU.

But how capable is that really when compared to a 5090 or similar?

  • permalink
  • fedilink
  • source
  • hideshow 15 child comments
    •
  • [–] 46 points 2 years ago* (13 children)

    The 5090 is basically useless for AI dev/testing because it only has 32GB. Mind as well get an array of 3090s.

    The AI Max is slower and finicky, but it will run things you'd normally need an A100 the price of a car to run.

    But that aside, there are tons of workstations apps gated by nothing but VRAM capacity that this will blow open.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] 23 points 2 years ago (7 children)

    Useless is a strong term. I do a fair amount of research on a single 4090. Lots of problems can fit in <32 GB of VRAM. Even my 3060 is good enough to run small scale tests locally.

    I'm in CV, and even with enterprise grade hardware, most folks I know are limited to 48GB (A40 and L40S, substantially cheaper and more accessible than A100/H100/H200). My advisor would always say that you should really try to set up a problem where you can iterate in a few days worth of time on a single GPU, and lots of problems are still approachable that way. Of course you're not going to make the next SOTA VLM on a 5090, but not every problem is that big.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 7 child comments
    •
  • [–] 3 points 2 years ago* (5 children)

    Fair. True.

    If your workload/test fits in 24GB, that's already a "solved" problem. If it fits in 48GB, it's possibly solved with your institution's workstation or whatever.

    But if it takes 80GB, as many projects seem to require these days since the A100 is such a common baseline, you are likely using very expensive cloud GPU time. I really love the idea of being able to tinker with a "full" 80GB+ workload (even having to deal with ROCM) without having to pay per hour.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 2 points 2 years ago (1 child)

    Yeah, I agree that it does help for some approaches that do require a lot of VRAM. If you're not on a tight schedule, this type of thing might be good enough to just get a model running.

    I don't personally do anything that large; even the diffusion methods I've developed were able to fit on a 24GB card, but I know with the hype in multimodal stuff, VRAM needs can be pretty high.

    I suspect this machine will be popular with hobbyists for running really large open weight LLMs.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 2 years ago*

    I suspect this machine will be popular with hobbyists for running really large open weight LLMs.

    Yeah.

    It will probably spur a lot of development! I've seen a lot of bs=1 speedup "hacks" shelved because GPUs are fast enough, and memory efficiency is the real bottleneck. But suddenly all these devs are going to have a 48GB-96GB pool that's significantly slower than a 3090. And multimodal becomes much more viable.

    Not to speak of better ROCM compatibility. AMD should have done this ages ago...

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 2 years ago (2 children)

    This is my use case exactly.

    I do a lot of analysis locally, this is more than enough for my experiments and research. 64 to 96gb VRAM is exactly the window I need. There are analyses I've had to let run for 2 or 3 days and dealing with that on the cloud is annoying.

    Plus this will replace GH Copilot for me. It'll run voice models. I have diffusion model experiments I plan to run but are totally inaccessible locally to me (not just image models). I've got workloads that take 2 or 3 days at 100% CPU/GPU that are annoying to run in the cloud.

    This basically frees me from paying for any cloud stuff in my personal life for the foreseeable future. I'm trying to localize as much as I can.

    I've got tons of ideas I'm free to try out risk free on this machine, and it's the most affordable "entry level" solution I've seen.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago* (1 child)

    And even better, "testing" it. Maybe I'm sloppy, but I have failed runs, errors, hacks, hours of "tinkering," optimizing, or just trying to get something to launch that feels like an utter waste of an A100 mostly sitting idle... Hence I often don't do it at all.

    One thing you should keep in mind is that the compute power of this thing is not like an A/H100, especially if you get a big slowdown with rocm, so what could take you 2-3 days could take over a week. It'd be nice if framework sold a cheap MI300A, but... shrug.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 2 years ago

    I don't mind that it's slower, I would rather wait than waste time on machines measured in multiple dollars per hour.

    I've never locked up an A100 that long, I've used them for full work days and was glad I wasn't paying directly.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 2 years ago

    Exactly, 32 is plenty to develop on, and why would you need to upgrade ram? It was years ago I did that in any computer let alone a tensor workstation. I feel like they made pretty good choices for what it's for

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 2 years ago (4 children)

    … but only OpenCL workloads, right?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 3 points 2 years ago (2 children)

    Not exactly. OpenCL as a compute framework is kinda dead.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago (1 child)

    What types of compute can you run on an AMD GPU today?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 2 years ago*

    Most CUDA or PyTorch apps can be run through ROCM. Your performance/experience may vary. ZLUDA is also being revived as an alternate route to CUDA compat, as the vast majority of development/intertia is with CUDA.

    Vulkan has become a popular “community” GPU agnostic API, all but supplanting OpenCL, even though it’s not built for that at all. Hardware support is just so much better, I suppose.

    There are some other efforts trying to take off, like MLIR-based frameworks (with Mojo being a popular example), Apache TVM (with MLC-LLM being a prominent user), XLA or whatever Google is calling it now, but honestly getting away from CUDA is really hard. It doesn’t help that Intel’s unification effort is kinda failing because they keep dropping the ball on the hardware side.

  • permalink
  • fedilink
  • source
  • parent