In a truly free society, there would be no such thing as "public property". It would all be private, and usage of that road would have to be paid for by those that want to use it. And the owner of said private property would maintain that property such that it would provide value for all those that wish to pay for usage of it.
Did you ever consider that maybe those individuals on those houses don't use those roads? Maybe they walk, bicycle, or are plain hermits? (Odds are they aren't, but I'm sure there are a few). You want to work with absolutes, and then deny that aggression is present. Do you want to argue that taking money from someone in return for something they don't use is not aggression? How about a thief taking money from your wallet on the counter, and then handing you a sack of grain in return? Would you say that's not aggression?