For another exotic cane fruit, consider Goji Berry (Lyceum barbarum). It has purple flowers instead of the pale yellow gooseberry flowers, no thorns, and it does admirably where we are in Maine. I think the berries taste better when they're dried.

A different exotic fruit bush would be Haskaps (Lonicera caeruleae). Pale yellow flowers give way to long-ish sweet/tart blue berries that ripen around the time of early strawberries, before blueberries ripen. Bumblebees and mason bees seem to love the flowers, and the birds in our area haven't caught on to how tasty they are yet. Another exotic worth considering is Goumi berry (Eleagnus multiflora), which will perform anywhere that autumn olive does. I keep expecting the state to call it invasive but it seems to behave better than its cousins.

For rarer native fruits, I'd probably recommend looking at Amelanchier and Viburnum family plants. A. laevis is more tree shaped, but A. alnifolia is more bush-y. A. canadensis is sort of in between, being a multi stemmed shrub that can get up around 26 feet tall if you let it. V. lentago (nannyberry), V. cassinoides (wild northern raisin), and V. trilobum (highbush cranberry) are each wonderful in their own right.

If delicious and decorative is the goal, we always get comments about our purple flowering raspberries (Rubus oderatus) and they're naturally thornless. They will send up new plants in a radius from their original spot - I've seen some coming up some six feet away - but I wouldn't consider them aggressive spreaders. They just like to make sure the space gets filled