Winter was dedicated to all the maintenance stuff mostly, and there was mostly no other field work during that time. They had way more holidays as you say, and they also had a lot more breaks during the day. Historians have done studies on this. An average 8-hour labourer today works about 1800 hours per year, accounting for breaks and holidays. An average medieval peasant would work significantly less so. English peasants had it worse at 1600 hours. French and Germans would fluctuate at 1300-1400 hours. Italian and French would also fluctuate at 1200-1300. Byzantine peasants (whose majority were not serfs and worked on their own land) would work much less at 1000-1200 hours per year!