They just don't bother with supporting alternative operating systems and are really invested in Android. The non-Apple ecosystem is almost entirely Android focused and I assume integrated circuit makers help support the use of their devices with Android. And phone makers have a lot of people working on integrating those devices with their phone and Android. It's also extra work for phone makers to document their devices fully. And some integrated circuit makers don't publish the specs for their ICs (you have to enter into some deal with them) and so on. I also think a lot of Android device makers just view their devices as disposable. A few years of updates and then people are supposed to get a newer phone. So the device drivers and integration doesn't have to be maintained forever, just long enough for the support period.

And some of the integrated circuits in phones are just really complex, like the camera. So it's also a matter of developer labor to make it work with Linux.

I'm not really sure what the single biggest factor is in making it difficult.

It really sucks that these extremely sophisticated devices that take a long time to even manufacture, only lasting a few years. But I suppose it's the same with a lot of things.