Fairphone 4 user here too. I also got it second hand, but immediately replaced the OS with CalyxOS. It seems fast to me. My last phone wasn't.
My only problems with it so far is the lack of a jack, that I need to remove the battery to remove the SD or SIM card, and the screen seems too easily scratched. It was scratched when I got it, but not this mutch. Its in my pocket with nothing else, but the screen is basically full of micro scratches while my previous phone of 8 years has (which was a very cheap phone even back then) almost no scratches at all.