Not a book, but I immediately thought of a point and click game that does something similar. In A Date in the Park, you play a guy who moved to Portugal, but doesn't speak the language. He goes to the park to meet up with the woman he has a date with.
There are several key moments where the language barrier ultimately works against you. I first watched a let's play of the game, where several viewers translated what was being said. Really changes the mood of a seemingly cozy game.