It's an old idiom. To say something is 'inside baseball' is to say it relies on a rather esoteric or specialized knowledge about the field.

If you asked most people what the worst possible sin could be, you'd get things like murder, rape, etc... Only the people that have done their reading know that this is explicitly labeled such.