I ended up using the external nginx for setting rules for rate limiting and such, and there I see the real ip. It's before traffic even gets to mlmym so that's good.
Also made sure to make it use the internal Lemmy url and skip a lot of the proxy stuff I was using earlier. It seems faster now.