When I DM, I always keep the idea in my back pocket that an enemy that has been distracted by a familiar too often will ready an attack to get rid of it the next time it is in range.
It'll still eat their action, might miss, and I telegraph it sufficiently that an attentive player might adapt their familiar's behavior, but it's a thing that can mix up combat and keeps players on their toes.
