Is there a massive difference between the two? Anything I should be wary of with either of them before I use it? I'm definitely privacy focused and have been trying out both but have been wondering if Vivaldi with uBlock Origin serves my purpose just as well as LibreWolf would.
There are two distinct things to optimize for here: your immediate privacy, and the future of a non-corporate web.
If all you care about is the former, then do... whatever. But if you also care about the latter, you cannot use a browser that supports the Chromium monopoly. That means using any Firefox fork. Personally I use Firefox itself because it's Mozilla that employs a paid security team, whose work all the forks are freeloading off.
A web browser is the space shuttle of consumer software. The Gecko rendering engine alone has an enormous attack surface. Which of these forks is contributing security patches to keep it secure?