It's less the time, and more how it interacts with the rest of your live.

Like the bodies instinct related to light amount and how you allow your body to sleep getting out of sync.

Or how day time tends to be more noisy and as such is more likely to disrupt your sleep.

Or how meeting people fits less into your sleep cycle.

Or how most people are not night active all the time, but switch between periods of night shifts and e.g. not being night active on your free days (because you e.g. can't go to the SPA at 2am).

Or how bright lights, even such advertised as daylight lights, often do not replace day light properly.

Etc.

So, while the time doesn't matter directly in practice having a "off" cycle is much much more likely to affect your life implicitly in negative ways then a more day aligned cycle.