That's not how the complexity of the mining works. It doesn't increase with every block. It scales depending on how many miners there are (essentially). So if many people stopped mining, the complexity required would decrease.