Had this issue as well, didn’t want to forget the WiFi network since I used it often but wanted to stop the HomePod connecting.
In the end I ended up blocking my HomePod from the other network’s settings so it can never connect to that WiFi network.
If you’re friends with your neighbor and his router lets you do that, then go for it, I don’t see Apple changing this even if you file feedback.