In theory it is safe. When a Nix package is built it isn't "installed". Unless root is running/installing random packages out of the Nix store there is no problem. As long as the user's aren't added to the trusted-users option they shouldn't be able to cause any problems for other users.

However like any multi-user system you are sharing a Linux kernel. A kernel is a very complex piece of software with a huge attack surface. Privileged escalation vulnerabilities are commonly found. (This also applies to the nix-daemon, but it is a bit smaller attack surface but vulnerabilities are still occasionally found.) So you shouldn't assume strong security isolation. I would say that a setup like this is acceptable for mostly-trusted people like coworkers or friends that are not expected to actively exploit vulnerabilities but definitely wouldn't let random unknown users use the system.

So if you want strong isolation use a VM or separate hardware, but then you won't be able to share the builds and packages defeating the point in this case.