I can 100% guarantee you that the owners will just be extremely confused about the idea that people might not want other people's "pets" in their yard. I can 100% guarantee you that the response will just be "well it's a cat."
And it means that if something ever happens to this small animal, OP will get blamed for it if he says anything to the neighbors.