Ooh, I hadn't even thought about voltage being backfed through the neutral.

Have you heard of something called a floating neutral? That could cause some issues.