You might be surprised! Records-keeping in the high medieval period, when feudalism is most 'defined', can be frighteningly rigorous, and runaway serfs were regularly retrieved even at some distance from their land.

All the same, in support of your argument, there's a reason why the "year-and-a-day" was a common legal standard for shedding serfdom by residence in a city - after a certain amount of time lapses, it does become increasingly difficult for institutions with pre-modern communications and government institutions to keep trying to search for and punish folk for running off - and increasingly inconvenient to have some local lord's thugs nosing around one's employees!