but as society grows larger, managing slaves becomes less efficient

At least in Europe, serfdom (which is basically slavery barring some minor distinctions) only came to an end due to the plague, which limited the supply of able bodied workers so drastically, other lords began to offer to actually pay another lord's slaves/serfs to work their land.

Slavery in the rest of the world ended for various reasons, but a big one was moral objection, not that managing slaves become less efficient (AFAIK, anyway. The Confederacy in the US only ceased slavery due to being psychically stopped from a more powerful adversary. They would've continued, and indeed would've expanded the slave trade to new territory as planned, had they not been stopped. Remember, those merchants taking power were also gleefully profiting off of the slave trade, which was one of their 'commodities').

But If anything, slavery has only ever increased, as there are now more people in US prisons alone than there were slaves before the Civil War, and they're often doing labor for pennies on the dollar. And since private prisons profit from having more prisoners, they've even sued the state when they aren't happy with how little prisoners they get.

Of course, you’ll need more people to manage this land, so slavery was introduced, which in turn ended up with those who control all the agriculture being the top dogs.

From the archeological evidence presented in the book 'The Dawn of Everything' by David Graeber and David Wengrow; humans were able to sustain egalitarian societies as both hunter gatherer and after agriculture was discovered. It is only quite recently in our species existence that authoritarian hierarchies seemed to develop, and not necessarily due to an environmental need. There is no technical reason slavery had to develop, it was simply performed and enacted due to the benefit it gave the owning group.

There's examples of slave-owning hunter gatherers presented as well, with neighboring tribes finding that behavior disgusting and refusing to participate in it, even taking in escaped slaves from those slave owning tribes.

Highly recommend reading it if you have the time, it's quite a compelling book. (Alternatively one of the authors gave a very (very) condensed talk about it at a TED talk here, and a longer interview about it here at Novara Media).

But at least in my view, it's clear that the evidence we have at hand shows this mode of society we live in that's highly hierarchical, competitive, individually selfish to the extreme, and operates on artificial scarcity, is quite the aberration from the norm.