Yeah, there's a fine line here. The DM improvising something whimsical and funny on the spot can be enjoyable to everyone, but if the party is going out of their way to do their absolute best to derail and force DM to waste any and all prepared material they're just dicks. The DM is still a player in the game doing it have fun too and doesn't owe you a campaign that bends to all whims of the party without restraint.

Also, a good lesson for GMs is to try and write any prepared material in a way that allows it to later be reused if players manage to miss it. Just because the party didn't investigate that one cave with a goblin ambush in it doesn't mean they can't run into a goblin ambush later down the line somewhere entirely different.