No, if you were just worried about energy, you would be much less tired than by doing things manually. For example, moving a heavy object downhill would gain you energy, not lose it, and keeping a heavy object in the air would neither gain nor lose energy. it would only be lifting that is hard, and it would still be easier than lifting manually.

Edit: and according to Wikipedia, human muscles only have an efficiency of around 20%, so doing basically anything through magic would be 5x easier than doing it by hand.

This is like saying you would be tired after lifting a pallet with a forklift. Many magic systems are about using magic in the world around you, which only requires you to understand how to use it; the energy being consumed doesn't come from yourself.

Now you see why the people at Hogwarts have a big feast every other day. And I don't recall the more corpulent ones using that much magic