Edit: Surprised at all the vegans in this thread. I didn't think there were so many of you. I'm glad you care so much about animal rights, that you're willing to forego eating them and using products made from them.
If you're not vegan and have moral objections for this, maybe you should look at yourself first and all the animal abuse you sanction by eating animals and using animal products.
Did you know dairy cows have to be pregnant to produce milk? They're artificially inseminated throughout most of their lives. I hope everyone complaining about this also complains about ice cream and cheese. Or else they would be hypocrites who just want to blame others but never look at themselves.
All morality ultimately comes down to assertions like that though. Ethics aren't properties inherit in the universe that can be objectively measured like the laws of physics. One can construct ethical frameworks, like utilitarianism or deontology or such, as useful tools to help one decide what one should do in an unclear situation, but ultimately, the choice of what framework to use or the rules of that framework comes down to certain things just not feeling right to a given person, and other things feeling okay.