But, yeah, generally this is true and is something that I miss from my FTE days. (I'm in tech sales now, and our hours are whatever fits into our customers' calendars...sometimes that means I start at 11, other times I'm going to be on an 08:00 zoom and like it!)
When I had a job I rarely had to interact with others, it was pretty easy. I could wake up early on the one day a week or less I needed to do some silly meeting, handle it, and go pass out for a couple hours and come back refreshed.
When it's your daily routine, it's effectively impossible. I can try to sleep in on the weekends, but that's really about it to catch up on decent sleep.
The days I can sleep in until 10-11am I am markedly more productive and just generally happy. But those are very rare now. Just the hours from 6am when my wife wakes up and gets out of bed until 9-10am are crazy in terms of how much better sleep I get overall. The previous 5-6 hours almost are immaterial. I could go to bed at 4am those nights and not notice a difference between going to bed at 11pm.