It's honestly not the taste for me. It's easy enough to cover up with sauces. The real issue for me, anyway, has always been the texture. I really have a hard time with anything that's curdy. It just makes me want to vomit. Maybe I was exposed to too much bad milk at some point I don't really remember...not sure. And the more dense and dry the curds are, the more difficult it is to ignore.
That said I do enjoy the soft, silky kind of tofu. It's more like a gelatin than a rancid milk left sitting out for days kind of feeling.