Noon is the only easily measurable time point in the day that is always (approximately) 24 hours apart. Also, sunrise and sunset are always symmetrical to noon, even when the day length varies considerably. For setting up a time system, noon consequentially is the correct reference point. But yes, your day rhythm will vary relative to that across the year, but there will differences according to local (like latitude) and personal environments which will make sunrise not a good reference point for a time system.