That wasn't all of it. Carter got blamed for both Stagflation and the high interest rates his Fed used to get it under control. And of course the Iran Hostage Crisis and oil shortages. Then Reagan got all the credit for the economy the high interest rates helped save.

Of course, that was a factor, but not the only one.

Carter's handling of Iran was poor even accounting for Reagan's sabotage. The hostage crisis and corresponding breakdown of relations would never have occurred in the first place if Carter hadn't granted the shah asylum, leading the Iranians to believe the US was plotting to install him in a coup exactly as it did before.