The color exhibited by anodized titanium is due to the refraction of incoming light by the oxide layer on its surface. This works exactly like how it does on some butterfly wings. The surface isn't colored per se, but rather the nanoscopic texture on the surface interferes with the light hitting it and bouncing off to produce a reflected color. Anything that fills in, changes, or alters the reflective index of the surface will alter its appearance, like any grease/finger oils/moisture/whatever until it's cleaned off. (This is why anodizing titanium, unlike aluminum, does not require any dyes to produce a color. And you can influence the color depending on the voltage at which you anodize it, which is wild.)
This is just how anodized titanium works. If this annoys you, a larger piece of it is likely to annoy you as well.