You constantly radiate heat. The warmer you are, the faster you radiate it away. In space this is the primary way you lose heat.

In your living room you are constantly bombarded by radiated heat from all the objects that surround you, even if they're just at room temperature, which lessens the effect. In space, not so much.

Someone who knows better might chime in, but as far as I know the trope of rapidly freezing out in space is exaggerated. You would definitely freeze eventually, but perhaps not as dramatically fast as portrayed in The Guardians of The Galaxy for example.