if, on a modern gaming pc, you can get breakneck speeds of 5 tokens per second, then actually inference is quite energy intensive too. 5 per second of anything is very slow
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments