If you are addressing the devices directly by IP, then the destination addresses are correct in the packet headers. The devices will respond with packets addressed to the source IP in the packets they receive.
https://serverfault.com/questions/157970/gateway-understanding-packet-details-ip-and-mac-addresses
The reason this is working is because the devices are still attempting to communicate in a best effort scenario.
This may work using a network switch, depending on how much processing is happening.
The more devices in the routing path, the less likely the packets are to tech their intended destination.
The point of this networking is to create reliable communication between groups of networks. The further off script one goes, the less predictable the communication becomes.