This is true. But VLLM's quantization support (as outlined in the article) is quite primitive.

Thats not what its built for. Its made for maximum speed with many concurrent users, not cramming LLMs into a desktop rig.

If you are running an LLM for personal use, you are usually better off with a backend optimized for higher fidelity quantization, like MLX DWQs, exl3, ik llama GGUFs and stuff like that.

If you want raw speed on a desktop, use one of the ninfer project branches.

inari@piefed.zip: Are you a bot? You are posting multiple times every hour of the day.

If you ascribe any level of sentience or intelligence to a computer, you definitely feel much less dumb than you actually are.

Having used these extensively, I can say with confidence that they feel exactly as dumb as they are.

This really killed a wish to run local LLMs. So fiddly...