Why not? If you are looking to diminish someone's right you need a better reason than "I don't like it". If you are looking to restrict someone's rights you need to give a reason not the other way around.

Let me give you an example of what I would think would be a good argumentation. I think we should illegalize circumcision. While this would restrict a parents right to raise their children with the religion of their choosing, circumcision goes against the childs right to be free from bodily harm which trumps the parents right to raise their children in the religion they see fit. The protection of a third persons rights is in my opinion a valid reason to diminish someones right.