You're probably really far away from the VR Chat server. Try pinging Google or Cloudflare, which will tell you ping to the nearest datacenter (a rough estimate of ping caused by your local ISP).
Based on their numbers, you could probably expect 50-100ms to Google, and then add an extra 90ms to get from there to your VR Chat server.
My personal fiber connection gets under 2ms ping on Speedtest