I would venture to guess it had to do with noon. It would have always been easy to say sunrise, sunset or noon even before a clock or sundial were invented. Remember there were no aircraft flying through the timeshifts. The effects of time on long distance travel were negligable if noticeable at all. Communications also traveled slowly. Once technology introduced clocks, you could see how your noon no longer aligned with the sun a couple hundred miles east or west. Your clock would not match the place you are visiting as noon had hands both pointing straight up as where the sun would also be located at that time. Your question only becomes relevant when we get light speed communication like radio and telegraph.

"Good afternoon German friend how is the weather today?" "WHY ARE YOU RINGING ME IN THE MIDDLE OF THE NIGHT, AMERICAN FRIEND!?" Not a real conversation, but you can imagine.

The answer: people hate change.