you are viewing a single comment's thread
view the rest of the comments
[–] 11 points 5 days ago* (last edited 5 days ago) (1 child)

Yeah RTX 2080 8GB and 32GB of system RAM, I wrote a guide

https://lemmus.org/post/24235317

I'll have to do an updated guide once Llama.cpp v0.7.0 comes out because there's a new option now for MoE expert caching. It doesn't work that well yet but I think they'll continue to improve it. https://lemmus.org/post/25955310

  • source
  • parent
  • hideshow 1 child comment