932
Anthropic logged net loss of over $80 billion in first half of 2026 (www.reuters.com)
submitted 1 day ago by to c/technology@lemmy.world
257 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] -4 points 6 hours ago* (3 children)

i would not recommend using the SSD to run local LLMS,its slow
best: VRAM only
good: VRAM + RAM
bad: Swapping

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 7 points 6 hours ago (2 children)

    I don't think they're insinuating running inferences from the SSD. But you need somewhere to store the weights for the GPU to load.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 0 points 3 hours ago* (1 child)

    i am trying to say: you need the SSD to store it but not swap to it due to its slowness

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 hour ago

    I won't pretend to be an expert but my understanding is that the architecture loads experts from the SSD on demand with the RAM serving as a hot cache and the VRAM running the active experts.

    As I said above I was able to run 70tok/s on my 4090w64GB ram and a samsung pcie4.0 ssd.

    I haven't run the benchmarks myself, but I can say the quality of response I was getting seemed on par with the frontier models I use for work (Claude opus/fable) and the published benchmarks for the model compete with those same models.

    It is a q3 quantization so there is some small loss of accuracy, but from what I've understood those are modest losses in the worst case.

  • permalink
  • fedilink
  • source
  • parent