I generally agree with you, but I've heard reasonable arguments like
- polygamy is patriarchy, gay marriage isn't
- incest is bad for the gene pool, gay marriage isn't
These raise their own questions of how to dismantle patriarchy, or if governments should have a say in our genes, etc. But I don't think they're equivalent discussions.