you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 2 weeks ago* (last edited 2 weeks ago) (5 children)

Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP

I bet you could run this in Q4 on 96GB RAM and 16GB VRAM, maybe even less. The benchmark scores seem good, beating 27b and DeepSeek Flash.

The n-gram embeddings sound very similar to Gemma 4 e4b embeddings. Need llama.cpp to support streaming n-grams from SSD, mmap would be less efficient than having explicit support.

GGUFs are starting to be available now

  • source
  • hideshow 5 child comments
  • [–] 2 points 2 weeks ago (1 child)

    I've gotten 35B-A3B running on a 4GB GPU with 16GB of RAM, the 6B actives should easily fit onto an 8GB card, but yeah, you might could get away with just 64GB of memory, maybe even just 32 if quantized small enough

  • source
  • parent
  • hideshow 1 child comment
  • [–] 2 points 2 weeks ago*

    the 6B actives should easily fit into an 8GB card

    That’s not how MoE models work. There are many “expert’ models and there is a static router model (dense) which determines which “expert” models to route the tokens through. What you want at a minimum is the dense portion of the model to be on VRAM and all of the weights to be in RAM/VRAM for best performance.

  • source
  • parent