I think that was the right action, but you could have explained better. Instead of just "Ok, you stay at the tavern" something like "Ok, you can stay at the tavern if you really want to, but you do understand that will mean you're sitting here bored all afternoon while the rest of us play, right?"