That's pretty much my understanding. Most of the advancements happened in memory speeds are related to the physical proximity of the memory and more efficient transmission/decoding.

GDDR7 chips for example are packed as close as physically possible to the GPU die, and have insane read speeds of 28 Gbps/pin (and a 5090 has a 512-bit bus). Most of the limitation is the connection between GPU and RAM, so speeding up the chips internally 1000x won't have a noticeable impact without also improving the memory bus.