You can make 1pm to 5pm as mandatory, so both the 9-5 folks and 1pm to 9pm folks have some overlap for synchronization, meetings and such.
As for people exploiting it, you should measure relative performance and not but time
One of my first startups I worked had a noon-4pm core hours. You had to be in by noon, and should stay until 4pm, the rest was flexible. I thought it worked really well for both early risers and late owls.