49
Consumer GPUs to run LLMs
(lemmy.dbzer0.com)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 02 Apr 2025
49 points (88.9% liked)
Selfhosted
60093 readers
754 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam.
-
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
-
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
-
Submission headline should match the article title.
-
No trolling.
-
Promotion posts require your active participation in selfhosting or related communities, or the post will be removed. No more than 10% of your posts or comments may be self-promotional, or your post will be removed. F/LOSS Exception: If your post is about a project that is completely open source & can be self-hosted in full without payment, your post is exempt from this rule as long as you continue to engage in comments.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 3 years ago
MODERATORS
Thing is, you can trade off speed for quality. For coding support you can settle for Llama 3.2 or a smaller deepseek-r1 and still get most of what you need on a smaller GPU, then scale up to a bigger model that will run slower if you need something cleaner. I've had a small laptop with 16 GB of total memory and a 4060 mobile serving as a makeshift home server with a LLM and a few other things and... well, it's not instant, but I can get the sort of thing you need out of it.
Sure, if I'm digging in and want something faster I can run something else in my bigger PC GPU, but a lot of the time I don't have to.
Like I said below, though, I'm in the process of trying to move that to an Arc A770 with 16 GB of VRAM that I had just lying around because I saw it on sale for a couple hundred bucks and I needed a temporary GPU replacement for a smaller PC. I've tried running LLMs on it before and it's not... super fast, but it'll do what you want for 14B models just fine. That's going to be your sweet spot on home GPUs anyway, anything larger than 16GB and you're talking 3090, 4090 or 5090, pretty much exclusively.