Nice to see you're getting close to the core of the problem.

Yeah, Land is owned rather than belonging to everybody as it used to be back before monarchs and as land ownership works in this system - any one individual with massive wealth can own way more Land than they need - it can be easilly hoarded by those with more money in a way that's impossible for those with less money to overcome (short of a Revolution) and thus create cartels or even monopolies in Land in desirable places to live, a market positions from where they can extract as big a rent as they want since everybody else has no alternative.

It's from the massive imballance thus created by Law in the main, essential and irreplaceable, "raw material" for housing that the massive house prices we see come from, and landlords usually use their priviledged position in that highly imbalanced market to extract excessive rents.

The whole situation is actually the very opposite of the "Free Market" you state it is - Land (and thus housing) ain't like teddy bears and soap were a competitor can just enter the market and make more of it when somebody tries to corner the market, quite the opposite: it's dependent on a naturally limited resource on top of which a trully ancient kind of legislation makes hoarding extremelly easy for those lucky enough to have lots of wealth, artificially transforming the limits of that resource into an extreme kind of scarcity.

In this Not-At-All-Free Market, most landlords will extract excessive rents far beyond the value they add. If rents were not mainly based on exploiting a dominant position in a market dominated by hoarding and artifical scarcity and only paid for the actual service being provided by landlords, they would be tiny in comparison with the current situation and very few people would be critical of landlords.