Skyscrapers are essentially a bunch of very heavy concrete+steel slabs with, comparably, weak spacers in between them. If one of the floors collapses on top of the other with enough speed/force to break the spacer it continues to fall unabated. Now it has double the weight and it's no longer static meaning that it's constantly accelerating due to gravity.
This cascades down every floor gaining increasing amounts of mass*speed(energy) and by the time it hits the foundation it's more powerful than any sort of "timed charge".
The amount of energy needed to start this cascade is several orders of magnitude less than what is needed to 'push it over'. By the time it's behaving under physics like the 'jenga tower' scenario you're envisioning it's <10% the height of the original structure.