I think it’s a little disingenuous to imply that deepseek is a small model. Even flash requires 160gb of vram and it’s important to note that’s a size aimed at a constraint, the vram density of hbm equipment from two generations ago. (And the density allowed by using last generations consumer equipment plugged into ten amp circuits with doubled vram see all those 48gb 4090s floating around!)

They didn’t arbitrarily decide that they’d use a smaller model, they were targeting a constraint. Which of course was really juicy because of all the demand for text inference and the limited new hardware to fulfill it.

The paper you cite is true, I haven’t read it but it based on the title it can’t really be wrong unless they just get wildly over their skis with their claims…

You’re right that everything’s gotta walk through the layers and more layers means a longer walk. That’s limited by memory and interconnect bandwidth though which is still doubling or close to it every year or close to it.

Which means it would have to be twice as fast to use a smaller model on cutting edge hardware to be a real “wall”.

We are not near the end of the memory bandwidth road yet, quantum tunneling isn’t rearing its head again as the workaday wafers that ferry serialized streams from place to place get faster and more numerous. Because they don’t need to get smaller really. The port of New York can grow and sprawl and sprout heretofore unseen support structures like coolers and voltage regulators and coolers for its voltage regulators.

I’ve seen in action what you describe though. On a, sensible chuckle, tiny card like a 3050 slotted into a system with pcie3 very small models run faster than those who can barely fit in the vram with their little bitty context, but that difference shrinks significantly when the card is a 5060 or something that still has limited vram but can load faster due to a faster pcie interconnect.

Now that may seem like apples and oranges because it’s two completely different things but the point of the comparison is to show that the difference between performance measures in time to first, tps or whatever other measurement might be in vogue at the moment when two models are compared shrinks when the interconnect and memory bandwidth gets faster.

To butcher a car metaphor, you can turbocharge an ls and get more power but you’re not escaping engine wear=crankshaft rotations. That might not be as butchered as it first seemed even though the domains are all shifted.