In my experience with man overboard drills clock faces are used as you describe. Port/starboard is only for low precision things.