Just think about it in terms of the number of people and number of days per year where they need to raise the indoor temperature by a certain amount, compared to the number of people and days needing to lower the temperature.

If you're looking at a place where it's 95°F during the day and 85°F at night, and you like to set the temperature to 75°F, you're only cooling it 10-20°F by time of day.

But if you're looking at a place where it's 35°F during the day and 15°F at night, and you want to heat things up to 65°F, you've gotta change the temperature by 30-50°F throughout the day.

Even when you're comparing absurdly hot weather to absurdly cold weather, you're still comparing something like 110°F to 0°F. You're still looking at a 70°F swing versus a 35°F swing towards comfort.

Throw in the fact that combustion of fuels (fossil fuels like heating oil or natural gas, or other fuels like wood in a fireplace) is usually only about 1/3 or 1/4 as energy efficient than the equivalent temperature change by heat pump, and you can see how much more energy intensive the typical indoor heating setup is compared to the typical indoor cooling setup.

Some of it is obscured by the fact that fossil fuels are much cheaper per unit energy than electricity from the grid, so that heating bills aren't as expensive in the same ratio, but in terms of actual energy used, it's a big difference.