I think it's weird not to use a skin colour emoji. An everyday form of prejudice is the assumption that white is the default. Black or brown is an exotic/diverse category. That other people have odd customs and social norms, but our customs and religions are normal.

It's even worse in design where the default human we design for is a man, hence women struggling with phone sizes that are too large for their hands. I think anything that helps break away from these default assumptions is good.

We already know from the Simpsons that yellow is a stand-in for white so it's not neutral. There's a white skin colour emoji, and by not using it when black/brown people use the other emojis, you're refusing to acknowledge that you too have a non-neutral, non-default skin colour.