The kind perspective is that drivers don't really have a vision for how much calmer and nicer things would be on their neighborhood streets if everyone drove 30km/h (20mp/h). They also don't really have a good perspective on how little the lower speed limit would affect their commute, and the safety upsides. And so without some more effective communication, they're going to perceive this as a straightforward degradation of their own quality of like (which is tied to their daily driving experience).

My unkind perspective is that drivers are maniacs who have lost the ability to cooperate and compromise with other people in a civilized setting. You can see this if you ever go to a town planning meeting about any road renovations that would add any amenities for non-drivers. It will be filled with surly car drivers who will at minimum disrupt the business of the meeting with angry outbursts, and at worse be on the edge of actual physical violence. The town meeting where we discussed adding a bike lane on the bridge that goes over the train tracks (to the literal "other side of the train tracks") was a very enlightening moment for me, as I had never seen my fellow citizens so political activated by something that I thought was obviously good. They were basically saying the quiet part out loud: that the type of people that would use a bike to cross that bridge are not the kind of people that they want coming over to their side of the community.

I've brought up the idea of a city-wide 20mph limit when talking to town officials after one of these meetings, and their response was basically "yes, that's a good idea, but it's completely politically infeasible because of the backlash we would get, so we choose to not even bring it up as an idea in these meetings so that they don't get too sidetracked".