Well that sucks. I'm trying to remember where my saddlebags drape, but they're definitely not in any possible line with the exhaust vent, nor are touching. I think my actual panniers made for the bike have a metal shield that sits between the exhaust and the pannier sort of hovers over and to the side of the exhaust.
Maybe your exhaust is just super special in how much heat it throws out.