And before empires and agriculture and such, people weren't really organized enough to steal land (and weren't embedded deeply enough in the land for it to be stolen). They'd just move to the new land and become part of the people already there.
Yeah, that's my point. At one point in time land wasn't something owned, just something utilized. The fact that governments exert control over them implies they were stolen, as it prevents some people from utilizing it. See: Proudhon - What is Property?