• *for certain definition of self hosted.

Developed by Chinese open weights model dev Z.ai, GLM 5.1 isn't exactly a small model. At roughly 754 billion total parameters (40 billion activated for each token generated), the model requires roughly 380 GB of GPU memory just to hold the weights and another 6-26 GB per user to maintain context depending on precision.

To serve GLM-5.3 at scale, Revolut deployed 56 Nvidia B200 GPUs, equivalent in GPU count to seven eight-GPU DGX B200 systems. But as Abitbol noted, the greatest performance gains didn't come from adding GPUs, but rather tuning the model's hyperparameters along with switching to NVFP4 quantization, implementing disaggregated prefill and decode, and tuning context lengths.

you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 16 hours ago (1 child)

The quoted numbers are the interesting part. 380 GB just for weights plus several GB of context per user means this only pays off at Revolut's scale. What stood out to me is that the article credits the biggest gains to NVFP4 quantization, splitting prefill and decode, and context tuning rather than adding GPUs. Those tricks carry over to much smaller setups too.

  • source
  • hideshow 1 child comment
  • [–] [S] 3 points 15 hours ago*

    Agreed. Economies of scale are about the only way this makes sense.

    I was just working thru the use case for purchasing a 64gb Strix (vs the typical coding I might do using OR or Open Code Go). The ROI is somewhere in the order of 7-10 years for me. So, on a purely dollar cost basis, local inference is $$$. OTOH, from a sovereignty and "own your own code" basis...there's something to be said for it.

    Probably I will run a few tests using OR with a few outstanding tickets of mine, to kick the tyres of Qwen3.8-27B. I've tried Qwen3.6-35B and frankly it's...not very good at code gen or patching.

    To boot, code gen isn't the only thing I use LLM for, so I suppose it depends on how bloody minded I want to be about future use case (I do actually have one).

  • source
  • parent