It's a bit more interesting than that...

Linux, unlike Windows, will let you delete a file that is actively in use. It removes the reference on the file system but the contents of the file won't be deleted until all file pointers to it close. In fact it will still be seen as taking up disk space until it's garbage collected (deleting a large file that's in use can be frustrating).

So the file is actually still available to that running process. If you replace it with a new executable and run that then you get the new version.

I bet this is done with inodes. Deleted files aren't truely deleted until nothing has it open and its inode gets dropped. Like ARC for files.

You can also do interesting things like overwrite a "file" (as in a specific filesystem path) with new contents while keeping anything that already has the file open on the old contents by unlinking the old inode to the path and writing the new contents under a new inode. I believe mv does this. The kind of lesser known feature that probably strikes a good balance between preventing super annoying silent errors/corruption and causing them.

Relevant Kevin Fang video

Wait until you see the Plan 9 API