llama.cpp in progress pull request for smart caching of MoE experts, 16% to 35% TPS boost for my RTX 2080 (github.com)

submitted 6 days ago* (last edited 5 days ago) by BeefAndPoultry@lemmus.org to c/localllama@sh.itjust.works

3 comments fedilink hide all child comments

git remote add miltos22 https://github.com/miltos22/llama.cpp-wackMall-merge-request.git
git checkout miltos22/master
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j $(nproc)
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 LLAMA_API_KEY=myapikeyhere ./build/bin/llama-server --models-preset ~/models.ini --models-max 1

In my models.ini file I'm using:

expert-hot-s = 10 for Gemma 4 26b a4b Q4_K_M QAT
- [56187] GPU hot store allocated: CUDA0, 1300296448 bytes (1240 MiB) for 10+1 slots (10 expert + 1 sentinel)
- From about 17 tps up to about 23 (+35%)
expert-hot-s = 32 for Qwen 3.6 35b a3b Q4_K_S
- [33181] GPU hot store allocated: CUDA0, 2422292480 bytes (2310 MiB) for 32+1 slots (32 expert + 1 sentinel)
- From about 25 tps up to about 29 (+16%)

This is basically a smarter version of n-cpu-moe. You could also set it to -1 for auto fill, but I don't think that works with GGML_CUDA_ENABLE_UNIFIED_MEMORY=1. I could probably push these numbers a bit higher, but I like having Chrome open.

Performance varies, and it takes time for it to adjust which experts are loaded into VRAM. You'll see lines in the log output like:

[40117] === Expert hot store: re-sync swapped 5 expert slots ===

You'll want to keep llama open for multiple prompts for it to learn which are the hot experts, and it can vary by task. It doesn't seem to ever hurt performance. There might be a bug in the Gemma handling, I saw it fail tool calls a couple times and I don't think it ever did that before, I'll revisit when this is merged.

In the pull request there are more knobs you can play with looking in common/arg.cpp but I haven't tested them:

--expert-heat-decay F                   expert heatmap decay rate per update (default: 0.999)
                                        (env: LLAMA_ARG_EXPERT_HEAT_DECAY)
--expert-heat-log-period N              expert heatmap log interval in updates (default: 100, 0 = off)
                                        (env: LLAMA_ARG_EXPERT_HEAT_LOG_PERIOD)
--expert-hyst F                         expert hot store hysteresis ratio (default: 1.3, 0 = off)
                                        (env: LLAMA_ARG_EXPERT_HYST)
--expert-dwell N                        expert hot store minimum dwell updates before swap (default: 0 = off)
                                        (env: LLAMA_ARG_EXPERT_DWELL)
-ehs,  --expert-hot-s N                 -1 = autofit slots from free VRAM, 0 = disabled, N = manual top-N
                                        slots
                                        (env: LLAMA_ARG_EXPERT_HOT_S)

Right now this is only for CUDA. Someone try this for DeepSeek V4 Flash 0731

top 3 comments

sorted by: hot top new old

[-] hummingbird@lemmy.world 7 points 6 days ago

Need to wait for AMD support.

[-] BeefAndPoultry@lemmus.org 3 points 6 days ago

In a few minutes a significant performance improvement incoming

👀

[-] BeefAndPoultry@lemmus.org 1 points 5 days ago

(Oops I got my Gemma and Qwen speeds mixed up, edited the post to fix it.)

But now with the new commits they added, with the same number of hot experts, Qwen is up to about 34. If I increase hot experts to 48 then I get around 37.

Gemma is still around 23 with just 10 hot experts. With 16 hot experts I get about 26 TPS. If I overprovision my VRAM (thanks to GGML_CUDA_ENABLE_UNIFIED_MEMORY=1) then 24 hot experts can give me 29 TPS, and 32 hot experts 34 TPS.

this post was submitted on 05 Aug 2026

27 points (96.6% liked)

LocalLLaMA

5007 readers

37 users here now

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

founded 3 years ago

MODERATORS

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Smokeydope@lemmy.world

MonsterBug@sh.itjust.works