There seem to be two main arguments put forth here:

1. F-Droid does not thoroughly audit the apps it distributes, so they might include bad behavior that is not initially obvious.
2. It is theoretically possible to provide a package to F-Droid that does not match the source code it claims to be based on.

To which I respond:

1. No app store thoroughly audits the apps they distribute. You must ultimately decide if you trust the developer enough to run their app, or audit the code and build it yourself.
2. This creates a theoretical opportunity for a developer or maintainer to upload a package that doesn't match its purported source code, but it's possible to check for this manually, and to automate that process. It's likely anyone exploiting this would be caught and their reputation tarnished. It comes back to the first point: do you trust the developer or maintainer enough to run their app?

If you have average security needs, you probably don't need to worry about this. If you have reason to believe someone well-resourced and dangerous wants to compromise your phone, you should probably be extremely selective about what apps you install and where you get them.