you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 2 weeks ago*

It's MoE so you can use --n-cpu-moe

https://lemmus.org/post/24235317

Low number of active parameters (6B) means you don't need much VRAM to get decent speeds

  • source
  • parent