Regarding the "no good reason to remove it," my understanding was that Samsung opted to remove it to increase the water resistance rating. Unfortunately the fairphone 4 only has an IP54 rating, so that's certainly not the case here.
Apparently they address the headphones jack question in an article on fairphone's support page.