Can't say I've actually seen anyone complaining about Stardew Valley or Animal Crossing for those reasons, seems like everyone loves Stardew, and AC complaints are just about nintendo's content pipeline being sparse on release.
That said, if people are complaining about "doing tasks" in Stardew and AC, I'd imagine that it's because it's mostly things you can do IRL. I can decorate my place or plant flowers or talk to other people in real life. I can't go kill bad guys in real life. In a way casual cozy lifesim games make me feel self-conscious, that instead of improving my virtual life, I could instead be doing this same stuff for my real life. Games that are more detached from real life don't seem as likely to trigger this guilt.
Also I'm fully expecting someone to respond that owning a house/etc is a fantasy, which is the fantastical appeal of Stardew/AC.