B0rax comments on FitMyLLM — Independent benchmarks for self-hosted AI

submitted 3 months ago by anzo@programming.dev [M] to c/homelab@programming.dev

5 comments fedilink hide all child comments

Check what can you use and at what rate of token per seconds would it be... It has examples of many models and quantization levels. Huge resource!

you are viewing a single comment's thread
view the rest of the comments

[–] B0rax@feddit.org 1 point 3 months ago (1 child)

What do you mean by „small gpu“?

I have not yet tried that, do you have any guidance? Or does „small gpu“ still mean >500€ GPU?

parent

[–] nutbutter@discuss.tchncs.de 1 point 3 months ago

By small, I mean GPUs like outdated ones, laptop GPUs, or like GPUs with only 4GB or 6GB of VRAM.

parent