40
Hardware for local inference? (lemmy.ca)
submitted 4 months ago by to c/selfhosted@lemmy.world
27 comments fedilink hide all child comments
 

I want to host some LLM's locally and use more advanced models. Since new hardware is out of the question, I think I should be able to pull something off buying some yesteryear equipment on ebay etc. Did anybody attempt such a project? Does it scale horizontally? (I.e. can I connext two boxes to overcome single box slowness?)

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 4 months ago (1 child)

Which is quite impressive to be honest, these machines were fucking expensive, and yet sold out completely

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 3 points 4 months ago

    512 gb would future proof you to run any local LLMs for quite a while. The speed at which they did it wasn't exactly bad too afaik due to uram being so fast. Dunno what other setup would compete here for the price.

  • permalink
  • fedilink
  • source
  • parent