There is economic conditions and social conditions behind it which deincentivised going back to "where they came from", because for one following their abolition they weren't given the farms to use as their own, the wealth they got were less than others in their own field even after, and they couldn't get proper representation until decades after

Plus, the African Americans had lived a life entirely within American territories itself, by now it was their own culture, they became not one defined by where their ancestors were bought and then abused but by how they survived this abuse, developed and grew a syncretic culture in their new homes

This meant that had economic conditions not been as bad, them leaving would've not been a wish to return to where they came from and reunite, because for them there was nothing to reunite with, they were separate people culturally and possibly even linguistically