It's not even about graphics alone.

They're clearly building their games in an extremely inefficient way. Starfield does not have anything going on in it that other games with much lower requirements also have done.

You see evidence of this in their previous games. One of the major performance issues with Fallout 4 for example, was that instead of building their cities in performant ways, they literally plonked every building as an individual asset into the world which thrashed the CPU for no reason. Modders just had to merge them all into one model to significantly improve performance. Their games are full of things like this and Starfield will be no different.