Depends heavily where you are and what community your part of. Some consider midget to be equal to removed. While others consider dwarf to be the bad. Over all midget is generally more prevalent as the term considered to be frowned on. But it is far far from universal.

As far as a fantasy world is concerned, if your setting has dwarfs then you likely should consider half breeds and how the culture would segment between humans and dwarfs. How they mesh together. And how abnormally short humans or tall dwarfs would be seen as out liers and how the lack of culture heritage could cause them strife or acceptance from the culture they most resemble.

Judging a book by its cover is a common problem for any group of people, and is the first step to over come in many fantasy stories. But proving the stereotype wrong you can flesh out characters, and show moral and social spectrums of gray. After all no peoples are black and white.

A abnormally short human could be seen as insultive to the dwarfs if he tries to integrate because he's seen as a fake. Since he won't have any true dwarven blood.

How he integrates into that society how it pushes back and how it eventually accepts or rejects him are all powerful themes and can mirror real life struggles to join a new community where you have no true roots.