91
Do you host your own AI?
(aussie.zone)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 24 Jun 2026
91 points (81.0% liked)
Selfhosted
60093 readers
950 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam.
-
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
-
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
-
Submission headline should match the article title.
-
No trolling.
-
Promotion posts require your active participation in selfhosting or related communities, or the post will be removed. No more than 10% of your posts or comments may be self-promotional, or your post will be removed. F/LOSS Exception: If your post is about a project that is completely open source & can be self-hosted in full without payment, your post is exempt from this rule as long as you continue to engage in comments.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 3 years ago
MODERATORS
Looping was a problem after reaching a certain context window size. The llama.cpp flags - -flash-attn on and looping penalties helped.
Probably that plus a higher quant solves it. Thing is most of us default to Q4_K_M as "precise enough"... and that seems to be kryptonite for the new Qwen's.
That's another thing with hosting AI that's not often discussed. Sure, you can maybe run that 27B model...but if it's at Q3_XS it's going to be .... "mentally challenged".
I've heard the Gemma models with QAT are meant to be near full precision at Q4 size. Haven't tried em yet.
Actually, on that topic - I've heard there's a different architecture (RWKV), that’s supposed to be much more efficient for long context because it uses an entirely different KV system.
Sadly, there are few RWKV native models and retraining a standard transformer to RWKV seems like a pain in the ass. I'd need to hire a cloud GPU, distill into a different architecture, mess with datasets .... honestly ICBF.
Yeah, a higher quant would be nice, I actually try not to go below Q5, but you can domino’s so much with 16GB of VRAM and the ddr4 system RAM.
But I must say I‘m pretty impressed by Qwen3.6-35b, not only from its capabilities but also from hardware requirements. MoE for the win I guess.
RWKV sounds interesting, have to look into it, thanks!