I actually did this once. The party was already pretty chaotic, but in a mostly fun way. They didn’t devolve into murder hobos, but they started to walk a route that made me more and more uncomfortable as a DM. So I put him into the game as a random one time quest giver.

It was a warning and the group understood it. Because the table suddenly got really quiet.