Their estimate of energy uses was only based on FLOPs, but I'd assume for real world energy usage the KV cache would be very impactful if not eventually dominant. It's probably also a bit unfair of them to ignore the Internet traffic and likely all the extra network traffic behind the load balancer.
Not a fan of their analysis, but I wonder if it's potentially close to accurate to this deployment? I can't imagine they're having large contexts and ballooning caches on a model meant for a phone.