Not everyone have to check something. But there are people that do routinely check popular stuff, either on their own or for their job. Sometimes this raises issues, which are usually handled appropriately. Of course if you download a little unknown piece of software made by a single person and never advertised anywhere, you'll have to do the job yourself. But anything semi-popular attracts enough attention to get some level of audit, at least because business uses a lot of open source. There are even businesses whose main product is auditing and developing open source, kind of like bounty hunters.

And of course there are counter-examples, too. TrueCrypt got pulled out quite dramatically, and I'm not sure we know why even now. But the more sensitive the stuff, the higher the chance of it getting some level of investigation.

As a software user, you can either care about your privacy or not. Caring about your privacy and not either vetting what you're planning to use or checking that someone else has before using it, is akin to sticking your hand in a fire to find out if it's hot.

Taking that analogy further, malicious open source software is kind of like a burning building. It only takes one person to raise the flag for it to spread pretty quickly through social media or other means that it is malicious. The whole community doesn't need to acknowledge the fire for something to be done about it.