I think if you've done any sailing then it makes sense to have a different term for boat directions vs relative to you direction.

It's not like driving a car where you're always facing and traveling forward. A boat rarely travels exactly where it is pointed, and in some conditions the boat can be pointed in one direction and traveling in the exact opposite, also, as crew you could be facing any direction at any time. So if you need to relay directional information it's much easier and more efficient to use language which refers to the vessel and it's orientation.

If you say to someone "look right", they will, in most cases, look to their right, but if you tell some one to "look starboard" they will know to look to right of the boat as it pointed. Which, in an emergency, could mean the difference of hitting something and sinking your boat or being able to avoid it.