Basically the title.

For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.

you are viewing a single comment's thread
view the rest of the comments
[–] [S] 1 point 6 days ago (1 child)

Not necessarly, I'd say. We've seen many releases with multiple sizes and the 27B class has been around since at least the Gemma3 times.

  • source
  • parent
  • hideshow 1 child comment