552
Anon helps people on StackOverflow (sh.itjust.works)
submitted 2 years ago by [M] to c/greentext@sh.itjust.works
42 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 years ago (2 children)

Look into ollama and mixtral variants... You'll be limited by things like GPU memory though.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 3 points 2 years ago (1 child)

    Mixtral 8x7B seems fairly capable to me so far. It's just that I need to wait a few minutes for it to reply, given I'm running it on a 1st gen Ryzen...

    Any models you'd reccommend that fit into 4GB of VRAM?

    I've tried Deepseek Coder, and it certainly works well for quickly churning out bash scripts for whatever purpose I can possibly think of

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago (1 child)

    Similar story here with an old 10xx GPU. I've just started tinkering with dolphin-mixtral-8x7b, but it's early days.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 years ago (1 child)

    what'd you say a good GPU is for it?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago (1 child)

    An nvidia card with as much memory as possible - the newer the better.

    I'm also beginning to look into the dedicated accelerators like the coral, but at first wash, it looks like lack of onboard memory will be a massive bottleneck.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •