you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 5 hours ago

Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can't handle it.

8B models are a bit too dumb to use any tools reliably but should handle a basic conversation.
llama cpp will allow you to manually load a model and tweak settings.

unsloth will give you a basic setup with some tools, manage model downloads, and settings.

  • source
  • parent