I don't think that's true at all. The viewers found him annoying, but except for Tuvok, the doctor, and occasionally the Captain, the crew seemed to be good friends with him. Janeway even made him an actual ambassador