Isn't it short for 'of the clock'?
Up until 1900, clocks and watches were very rare, so if you ask someone what time it was and they looked at a clock or watch they would say 1 O'clock. If they determined the time from a sundial or a stick in the ground they would say the time is 1 in the afternoon
[video]______________________________[/video]