Out of interest, how do you compare/store in a map? Do you convert to a string, or a [4]byte array (for IPv4), or some other comparable/hashable type?
I think this post is right about net.IP, though I also think the solution they've reached is Lovecraftian. But I write Rust, too, and I'd take net.IP over Sockaddr any day of the week.
My personal favorite is probably the python ones. Only annoyance is difficulty of creating modified objects from existing ones.