A lot of calorie expenditure is determined by hormones, and for a typical person, resting metabolism is a significant percentage of total calorie burn.

Some people respond poorly to dietary interventions because they end up with lower resting metabolic rate. Sometimes it shows up in the form of a person who is always feeling cold or exhausted.

Others have high stress hormones, which throw all sorts of metabolic loops out of wack.

And other hormones affect appetite or satiety after eating, so the biological disadvantage (whether genetic or environmental or both) sometimes comes in the form of greater difficulty with portion control or willpower compared to others. That's not a direct effect on calories in versus calories out, but it does have an indirect effect on overall success with a weight loss program, and it's grounded in an actual physical/physiological/biological difference between individuals.