Problem: even with an alternative operating system, you still don't get security updates for the baseband firmware, and that thing is a huge remote attack surface that, if compromised, grants the attacker unfettered access to the entire phone.
Some new phones isolate the baseband processor from the rest of the system. Only the small independent phone makers like Librem use such a design, though.