I use VBR and adjust the quality slider until I cannot see artifacts. I don't do anything particularly special and maybe there's more that could be done.

I once heard of an approach where you remove all grain and reapply it live to reduce the bitrate. That sounds interesting but denoising usually results in quality loss and it will likely look pretty artificial. My tooling also does not support it, so I've not bothered.

If someone can recommend me a good encoder or tool I can try that is better than whatever comes with handbrake I'm happy to give it a go.