Hacker News Reader

Comment by KenoFischer | Hacker News Reader

Hi HN, please note that this is an active discussion thread in the Julia community. You are all more than welcome to chime in, but we do try to keep discussions as productive as possible, so if you do decide to comment, I'd ask that

1) You familiarize yourself with the actual proposal and the improvements that are currently underway and

2) Be kind

A number of people have put in an enormous amount of effort to try and get this right - please remember that they are indeed people.

View on HN

Is the telemetry available to users? I glady opted into Synchthing telemetry after seeing this page: https://data.syncthing.net/

When the data is available to the community, just like the source code, its a much easier sell.

The plan is to make aggregate usage data available publicly and potentially share more detailed usage data with individual package authors. The exact format is TBD since it'll depend on the quality of the data that we get (this is not active yet, except on the preview build). The raw logs will be accessible to core developers with a reasonable need to access (e.g. they're working on the infrastructure or running the analytics), but will not be public.

How about deleting the IP addresses within 48 hours like 1.1.1.1 and 8.8.8.8 do?

https://developers.google.com/speed/public-dns/privacy

We do have a limited retention policy for the package server logs we keep (which include client IP addresses). It's not publicly stated anywhere right now, but one reason why we need to keep IP addresses is for abuse mitigation. We have been hit in the past by users that do things like download large (100MB+) files from our package cache servers multiple times a second for days on end. This is a particularly easy case to catch (since it easily pops to the top of any analysis you'd care to run, across any timespan) but there are more subtle forms that require a longer time window of analysis (e.g. users that download once per hour, all month) that would be lost in the noise without the ability to see what's going on.

This comment is not meant to serve as an official policy, just pointing out one of the reasons why we can't delete IP addresses like 1.1.1.1 and 8.8.8.8 do; because the abuse vectors for a server that serves the community large resources is very different from that of a DNS server.

Most of the "abuse" we see is not malicious in nature, but is instead users that have some kind of very poorly-configured autoinstaller on a cluster. In the case of a catastrophic issue like the one mentioned above, we null-routed the IP address, reached out to the abuse contact for that IP, and worked with the user to architect a better system. Everyone is happy now, and we can continue to provide a high quality service for the community without breaking the bank.

How about hashing IPs? You could still see if someone were on your abuse list if abusers.contains(hashfn(req.addr)).

Hash of IPv4 address can be easily reverted because there is a limited number of addresses.

Doesn't help for two reasons 1) If the has has enough bits to be useful for blocking, it's trivial to reverse 2) Even if it did make the IPs anonymous, we want to be able to email the NOC at whoever is sending the abusive traffic, so they can go investigate

> we want to be able to email the NOC at whoever is sending the abusive traffic, so they can go investigate

If you block their traffic with HTTP 429 Too Many Requests, they can email you instead.

We prefer not to break researchers' workflow because the group next door misconfigured their server. Happens all the time. We only sinkhole IPs if the traffic is malicious or on track to exceed or budget.

I don't work on this particular thing, so I can't say precisely what the planned retention period is. I suspect 48 hours is too short, since people do take weekends ;). It'll probably become clear with experience what retention periods work. DNS servers are in a very unique position of course since they essentially get your browsing history.

Since the data is not being made public, presumably it is judged to be sensitive to some extent?

So if follows then that users are right to be concerned and would have every reason to not opt in if they were presented with the choice.

What's sensitive or not depends very much on what other information the entity doing the analysis has available. Of course raw log records are more sensitive than aggregate data. For example, if somebody is wiretapping your internet connection, then even if the connection is encrypted raw logs would let them draw conclusions from timing. To some extent you're trusting the Julia project (or at least the people who have access) to not clandestinely be in the wiretapping business, but then again you're already trusting it with arbitrary code execution on your machine, so if it were in that business, you'd have bigger problems ;).

In any case that's why it's important to be transparent about what is sent, and for what purpose and who has access, so people can make informed decisions. Ironically, I think people are jumping on the authors of this particular piece of functionality precisely because they tried to be very transparent.

Yes, they are transparent about deciding not to offer the user the choice in a straight-forward upfront way (i.e. opt in) because "the vast majority of users will not opt-in". In other words, deciding that what the users want is not as important as the marketing stats.

And, as you say, users trust the developers with access to their systems and data. Deciding unilaterally to sacrifice user privacy to benefit other interests might be seen as a breach of that trust.

It’s actually more that “the vast majority of users do not care” and so go with the defaults.

An opt-out is the way that makes sense.

Reply on news.ycombinator.com