you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 hours ago (1 child)

Sabata, goddess of computing, can I run it on a 32GB system with a 4GB nVidia GPU?

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 1 hour ago

    Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can't handle it.

    8B models are a bit too dumb to use any tools reliably but should handle a basic conversation.
    llama cpp will allow you to manually load a model and tweak settings.

    unsloth will give you a basic setup with some tools, manage model downloads, and settings.

  • source
  • parent