Depends entirely on whether or not Moore's law has hit a physical limit yet.

If we're still doubling computational every X years for the same cost, we'd be able to see same models cheaper. If we're however finally hitting physical limits, then yeah. It won't be cheaper and we can't just smash more context in it like there's no tomorrow.

Then the most likely breakthrough would have to be DNA storage (tri point storage, rather than bi point storage) and then we'd need fast read/write to DNA storage. As that'd theoretically allow for more context for the LLMs, context is the biggest bottleneck