Papal infallibility is about what the popes teach, not about what the popes do. There are some controversies about whether popes in the past have taught things that are contradictory, such as John XXII with the Beatific Vision controversy, or Honorius with failing to reject Monophysitism (which opponents of papal infallibility would strengthen to saying that the pope taught monophysitism, not just that he failed to reject it, based on his anathemization in the subsequent council of Chalcedon).

IMO this opens up an interesting question of to what extent Alexander VI and previous popes that allowed slavery (or actively gave orders that would see it expanded) were teaching when doing anything at all. But I don't think it's necessarily contradicting the understanding of infallibility that the Church has held since Vatican II, which means that this shouldn't bring anyone from believing in Catholicism to not believing in Catholicism (unless they're really, really racist... which actually maybe does include a lot of people)