Really they're designed to prevent the passage of heat from inside to outside, so that your heating (or in the past, your fire) didn't pointlessly heat the outdoors. But this design can't work only in one direction, so it also slows the passage of heat from outside to inside.

There are three main ways this slowing happens:

1. Any solid barrier prevents the passage of air from the environment to the place where you are, and vice versa. Obviously this is symmetric.
2. Cavity walls have an air-gap in them. Air is a poor conductor of heat, so while hot air on one side of the wall heats the wall's bricks up, this heat then travels only slowly to the other side of the wall. Again, this is obviously symmetric.
3. What you're talking about is that bricks and stones have a high thermal mass. This is a bit more complicated so I'll explain: it means that if you take a given lump of heat (say, all the heat coming off a fire for one minute) and apply it to the bricks so it's fully absorbed, they'll heat up less than if you were heating up a wall made out of steel. The energy is still in there, though, and when the temperature of the air next to the wall drops below the wall's temperature, that energy is released back into the air. But this too is symmetric: it applies just as much to the energy of a fire as it does to the energy of, say, one cubic metre of air at 35 degrees celsius. The same property that releases heat into the house on a winter evening releases heat into the outside air on a summer evening. This is what you want!

Each of these properties is symmetric, because physics doesn't "know" which side of the wall is inside and which is outside - it only "knows" which is hotter and which is cooler. So the exact properties which keep you toasty in winter help keep you cool in summer.