But why doesn't it ever empty the swap space? I've been using vm.swappiness=10 and I've tried vm.vfs_cache_pressure at 100 and 50. Checking ps I'm not seeing any services that would be idling in the background, so I'm not sure why the system thought it needed to put anything in swap. (And FWIW, I run two servers with identical services that I load balance to, but the other machine has barely used any swap space -- which adds to my confusion about the differences).
Why would I want to reduce the amount of memory in the server? Isn't all that cache memory being used to help things run smoother and reduce drive I/O?