Signal might be one of the most audited pieces of software in existence. Any criticism is likely either coming from or is supported by countries that fear encryption such as China, Russia and Iran.
The big downsides of Signal are that it requires a phone number and that is depends on Signals servers. That is it. You messages are completely safe as all messagers use the same underlying cryptography.