Don't get me wrong, if you have a good jam or preserves it's totally fine on a sandwich, and I wouldn't turn my nose up at it. However, peanut butter doesn't really need the help, and the extra sweetness from jelly is totally unnecessary. Toasted bread and peanut butter is all you need for a quality simple sandwich.
I always hated PB&J. Both were fine on bread (though toasted was preferable with the jam), but I hated them mixed. I think I tried it twice ever in my life. It's weird since I love all kinds of sweet + salty combos; I think it's a texture issue.