Is it because latency does not scale in that way?

Yes, your understanding is fundamentally flawed. Starlink adds a fixed latency on top, if you ping to a server was 2ms with fiber and 52ms with starlink, then your ping to a server that would be 100ms with fiber would be 150ms with starlink