From a personal freedom POV, I agree. But, if it was easy it would be a support nightmare.
Google and Apple scan every app that gets loaded into their app stores for malware. There's also a lengthy review process, even just for updates. Some malware does still slip through, but it's a trickle compared to what gets blocked. If sideloading apps were easy, my younger sister would be in so much trouble. She'd have various accounts phished within a day. She'd install something that drains the battery within an hour and not understand what was going wrong. And, she's relatively tech savvy. I have no idea how the older generation would survive.
Of course, since Apple and Google make 30% of every sale on the app store, they're not purely motivated to just keep their users safe. The real problem is that there is a duopoly in smartphones. Apple and Google have essentially the same policies, and if you don't like them you have no other options. If there were a dozen OSes, there could be smart phones for Granny that had everything locked down, and smart phones for h4x04z that didn't. Companies that struck a good balance between protecting their users and allowing their users freedom would do well in the market. Companies that didn't would shrink and fail.