This assumes a whole bunch of things. First, do you actually absorb all of the calories through your gut? Does your body maintain the same base rate of expenditure (BMR) in both the short and long term of restriction?
When you look at people who did The Biggest Loser they did the exact thing you are talking about. They had a significant caloric deficit through eating restricting and massive amounts of exercise. In the short term they did lose weight but it also ruined their BMR. Years later they had mostly put the weight back on and had a lower BMR than at the start. It damaged them.
If you lose weight through caloric deficit you may not notice any change but your body will stop prioritising things like your hair and skin, muscle growth, and bone maintenance. All of those are low priority for an organism in caloric deficit. Instead your body will focus on the most important thing, getting more food. You become food obsessed, thinking about it all day, and you will eat almost anything you can access. You also end up with a lower body temperature, less immune activity, and lower drive for exercise and sex. It is an absolute nightmare.
The end result is that calories in calories out assumes a perfect and simple system. It does not take into account things like proton uncoupling in brown fat, differing levels of absorption through digestion, body temperature, hair growth, cell turnover, and tonnes of other things. It can appear to work for a short time but long term it breaks down and deviates more and more from the data.