Huh, sounds like a very unreliable setup. Admittedly mine is much simpler and I refuse to use voice control for anything at all, but I experience zero glitches with my Shelly switches and HA integration.