I live just outside a no-fly zone around an airport. Maybe a mile away, slightly under.
I never take my drone above the trees unless I'm away from home because I'm paranoid about anything going wrong and my drone just taking off in one direction until it dies or hits something.
I'd rather be extra cautious and be lightly mocked than cause the airport to shut down and have the alphabet boys knocking on/down my door.
Especially since I live close enough to a military base that my drone could theoretically reach it in a flyaway, if winds are helping it along.
Note: I have been lucky enough that in four years, I have never lost contact with my drone, but the paranoia remains.
A person can dream.