Because the two clefs are symmetrical around middle C. Line "6" of the bass clef is line "0" of the treble clef. They are not two different clefs, the are two halves of the same 11-line staff.
> some of the time I'd be thinking about start/stop times rather than durations
It's actually a combination of both. Bar lines give you a bigger multiple, beams give you a smaller one (typically 1/4 or 2/4). Only within each 1/4 or 2/4 do you have to think in deltas.