777
Generative AI Is an Engineering Disaster. A shockingly inefficient trillion-dollar project. (www.theatlantic.com)
submitted 1 month ago by to c/technology@lemmy.world
213 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 1 month ago (3 children)

It's the q4 quantization, but it requires 20+GB vram and my 5080 only has 16

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 4 weeks ago (1 child)

    What are you using to run the model? Llama.cpp will automatically split the model between your system ram and graphics card's vram.

    Qwen 3.6 is a mixture of experts model with only 3B parameters active at a time. Even without quantization your card could easily run that.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 weeks ago

    It has been some time since my initial comment so at the time I was mainly using LM studio. Qwen 3.6 a3b is the MOE and it does work well on my card, but the dense model that is more intelligent/capable is the Qwen 3.6 27b which doesn't fit on the card and does get offloaded, but offloading cuts the speed down to like 1/tps.

    I have since found a version of the 27b model that is "quantized," for lack of a better term, differently and has to be run through TabbyAPI which gets back to 30ish tps. It can't offload so it must fit fully on the card which keeps the speed high. Might be worth a look if you're interested, the only downside is that with my 16gb card the context limit has to be kept pretty low ~40k if I remember correctly

  • permalink
  • fedilink
  • source
  • parent