Show HN: Measure downloads and commercial adoption of any file you distribute
about.scarf.sh
about.scarf.sh
"All without ever having access to personally identifiable information or invading the privacy of your users."
Can you elaborate on how you go about that?
The main way this is achieved is by purging any personally identifiable information from our system, mainly the IP address of a download request. Scarf uses the IP to look up metadata like company affiliation, cloud provider, course grained location, etc, to surface that to you. Once that metadata is looked up, the original IP address is discarded. All information stored long term is fully anonymized.
Our goal is to help enable OSS developers to financially support their work. Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work? If so - companies are tracking people all the time at a very granular, personally identifiable level. Why should we hold OSS developers to an even higher standard than what we tolerate from large companies?
The problem here is that i DON'T tolerate this from large companies either. I find the pixel tracking thing outrageous and disable images by default in my email client to avoid it.
I understand your argument, I just find it personally strongly disagreeable, and I'm willing to bet poster above did as well.
If you are already using OSS today and grabbing that software over the internet, you are tolerating it even if you claim otherwise. If you pull something down from GitHub, Microsoft has all the data that we're talking about here.
> I understand your argument, I just find it personally strongly disagreeable
Fair! And I understand yours too. I also think your argument is more idealistic than practical for the current state of the ecosystem, especially considering how many parties already have access to this web traffic data. Maintainers having this data too is a very benign additional party to have access to it. Furthermore, it's a concrete way we can all chip in to help OSS maintainers and make their jobs a little bit easier, short of reaching for your credit card (which we should all be doing too).
> Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work?
No, but I shouldn't have to worry about that as a user. The onus is on corporations to disclose the software that they use in accordance with their respective licenses, the regular user doesn't deserve to suffer for the incompetence of funded organizations.
> Why should we hold OSS developers to an even higher standard than what we tolerate from large companies?
You don't, they do. That's the point of open source licensing in the first place: defining what you're comfortable with other people using your software for. By choosing an Open Source license, you're assuming one of the most difficult and thankless positions in the world of software. That's how it's intended to be though, because that kind of transparency is imperative when we're distributing free software. You wouldn't poison the rations being donated to the homeless, so why are you comfortable poisoning the CDN of my download? This all seems pretty cut and dried to me.
sigh Time to start dropping Scarf URLs in my hosts file...
If you are using open source today, you're already hitting servers that have access to all of the same information Scarf sees. Visiting a URL is by definition asking a server on the other side to process your request. That data can be very helpful to all of the great open source maintainers out there, but has historically been difficult or impossible to access. The result will be better informed maintainers, and better OSS for everyone.
End-user privacy does not need to be compromised in order to give OSS maintainers a basic quantitative understanding of how their software is used. This is our best attempt at a solution. We will be continually improving it better however we can.
(I should add that your sibling comment says they're using browser headers, which probably reduces this issue a lot)
At Scarf, we aim to give open source developers more visibility into how their software is being used. As people with experience distributing binaries and artifacts hosted on platforms like GitHub Releases and S3, a repeated struggle was not having any visibility into downloads. Which versions of the software were being downloaded the most? On which platforms? Where in the world? Which companies were downloading?
This year we built Scarf Gateway, which acts as a redirect/analytics layer for any container registry. Supporting other kinds of artifacts was a natural extension, and arbitrary file downloads is perhaps the most general extension we could build!
Curious to hear what people think.
How would someone opt-out of being tracked that something's been downloaded?
However, you seem to have an incomplete understanding of GDPR judging from your homepage. For example, you don't provide a way for people to opt out on your homepage. This may indicate that you are thinking about GDPR in American "PII" terms instead of thinking about "processing purposes" and "personal data" (not necessarily identifiable, such as a 5-star rating for a taxi driver) as intended by GDPR. You can store my home address without my consent if you need it to deliver a book to me. You may not pass my non-anonymized IP address to anyone except your secops (legitimate business need has been explained by EU courts to mean a need to fulfill user's need, not company need, e.g. to show ads).
Further down the thread you also discuss the opt-out mechanisms. Again, this is only legal under GDPR for opting out of the kinds of processing you have a legitimate business need for. Things that require a consent may not be worked around with an opt-out.
Not a lawyer but a person in EU who sent GDPR requests and complaints to company DPOs and regulators. Hope your service grows well!
Fully complying with GDPR is a requirement as we build this out. Our data policies and practices have been thoroughly reviewed by our legal team. If we are doing anything incorrectly with respect to GDPR, it will be promptly addressed.
It turns out that the data we are actually storing about end-user traffic do not meet the criteria that trigger requirements for explicit consent. Scarf also operates a data processor with respect to GDPR, rather than a controller.
The idea of this being an enhanced link shortener is a good metaphor. I think if were ever encountering an image or a download that's not going through a dockerhub or GH link, I might give pause and think about it a bit.
I'm not defending enterprise usage of packages, especially somehow if it's core to the business, but I don't need anyone to come knocking either.
Glad to hear you like this better, we do too. We built Scarf Gateway in a large part due to the response of the react-query community (and a handful of other projects) to scarf-js. Lots of discussion on GitHub and the Reactiflux discord provided good learnings for us: mainly that mechanisms that phone home, especially at unexpected times, were particularly unpopular. We also heard more acceptance of the idea that the registry/host platforms who already have this information could be sharing it with maintainers.
We want to support maintainers with better data in a way that best suits the OSS community and respects privacy. And so we went back to the drawing board, and Scarf Gateway is the result!
Still, I understand you may still have remaining hesitations here anyway with Scarf-powered download links. Are there any specific privacy concerns we can mitigate?
Avi gave me a demo last week as we’ve been looking on how to get better analytics for our open source framework IHP https://ihp.digitallyinduced.com/ it’s quick to set up and they also provides tools for doing analytics for eg the documentation.