It's not completely baseless. You can't get sick from the cold itself, but lower core body temp does weaken your immune system until you warm up, making it easier for you to get sick if you do get exposed to something.

In the US, I hear this more when your hair is wet: "Don't go outside, it's cold and your hair's wet, you'll get sick!"