Your ability to also tell that voice "but of course that's ridiculous for a large number of reasons" is the governor that prevents the voice from being a problem in most people who do have an inner monologue. When someone thinks their inner monologue is something else's voice, like a diety, that's when it becomes a problem because it removes personal agency while elevating the significance or validity of their thoughts.