This is incorrect. One of the most quoted phrases in case law goes something like "Justice must not only be done, but must also be seen to be done." (Rex v Sussex Justices)

The judicial system has a duty not only to be just, but also to demonstrate to the public at large that it is being just. That's why even if no wrongdoing has occurred, if there's a way people might think it has occurred, the justice system has failed.