Most of the people commenting appear to have no idea how Unix memory management works. You are spot on. You should worry about it if it actually has a noticeable effect.

Looking at memory utilisation doesn't tell you anything useful if your system is running normally.