Abnormalities from "normal" were a critical self defence feature, for our ancestors. E.g. a lack, or change, of bird song might indicate a predator in ambush. Unusual lighting might indicate a storm coming in.

Our brains are wired to learn normal patterns. When those patterns change completely, we are fine with it. When they change subtly we don't like it.

The threshold for this is different for different people. Personally, I'm fine with completely different maps, but off put by modified real maps. I also cannot watch soap operas, they are too close to "real" and trip alarms at their mismatches. Conversely, sci-fi and fantasy are fine, they are different enough to not set off my alarms. I know others who are set off by sci-fi, but soaps are within their norms.

I think a lot of it is humans are used to maps formed by tectonic plates shifting, glaciers forming and melting, storms and other weather, etc... When it's just an RNG heightmap it's missing all those familiar features like rivers, mountains, and dry lakebeds