you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 hours ago

I won't pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.

As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.

I haven't run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.

It is a q3 quantization so there is some small loss of accuracy, but from what I've understood those are modest losses in the worst case.

  • source
  • parent