Basically the title.

For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 6 days ago (3 children)

27B dense was a good size for 3.8, but I'm guessing they're going to add a few B parameters in order to make it noticeably better.

  • source
  • hideshow 3 child comments
  • [–] 1 point 6 days ago*

    If they can make the KV cache size scale more efficiently with many tokens, like DeepSeek 4.1 Flash fitting 1 million tokens in 890MB, it might give them enough room to add a couple billion parameters of weights and still fit well in 32GB GPUs

  • source
  • parent