Getting 10TB of GitHub logs and extracting details of all users and repositories | Hacker News Reader

Getting 10TB of GitHub logs and extracting details of all users and repositories

trickest.com

103 pointszaric50 comments

I'm a big fan of neat statistical analyses, but when I look at the archive site https://www.gharchive.org/ , the overwhelming feeling I have is "creeped out". Taking periodic snapshots of repositories and their issues and wikis sounds good, but do we really need a log of every time someone watches an issue, and every commit message being irrevocably set on public record? That level of details on individual activity seems to have little value outside of cyberstalking

I did have a look at the bottom of the page where prominent uses are listed, but nothing stands out as actually useful tbqh

> That level of details on individual activity seems to have little value outside of cyberstalking

A selling point for business-level is that they can do whatever OKR they want on top of that details. To make any employee dance whatever business can imagine. Any amount of Tb seems OK until it helps selling "Business" tier github.

I have a GitHub account under my real name, but recently I've started using GitHub under a couple of other names instead. There's so much stuff you do in public on GitHub that I want to avoid people doing exactly this kind of analysis on.

I wish using multiple identities was at least some level of foolproof though. I have to be careful to configure my local copies of repos to use the correct username, masked email, and PGP signing key. It would be super handy if git had a global config of multiple identities and if I could have it prompt me to select one either when I clone a repo or the first time I push to it.

Private repos are still off limits for this sort of analysis correct?

Microsoft owns github so draw your own conclusions on whether that's true and how long it will last if it is.

They should be, but those aren't a solution for anything where you want to share, collaborate, or even just ask questions on.

You can have includeIf sections in your .gitconfig that applies only to things within a certain directory. So you if you create top level directories in your $HOME for your various identities, all you need to do is to make sure you are cloning into and working within the appropriate directory hierarchy.

I do this to separate personal and company repositories on the same machine, and it works flawlessly. An example config looks like:

~/.gitconfig:

```

[includeIf "gitdir:~/Personal/"]

        path = ~/personal.gitconfig

```

~/personal.gitconfig:

```

[user]

        email = {personal email address}

```

And you can have arbitrary numbers of "profiles" like this, as long as each is in their own directory.

Oh that's really cool! I'm absolutely going to use this technique. Thanks both of you!

note that this is technically against their TOS if not using paid accounts:

> One person or legal entity may maintain no more than one free Account (if you choose to control a machine account as well, that's fine, but it can only be used for running a machine)

https://docs.github.com/en/site-policy/github-terms/github-t...

internet would be better with total anonymity.

That's one of my favorite parts of reddit. Accounts are just pseudonyms, and you can generate as many as you want. I personally generate a new one every few months, which helps keep too much identifying data from building up over time.

and they're all identifiably attached to your metadata package(s)

Which, at the very least, isn't public to everyone. So, not perfect, but better than single public account.

Not sure I follow.

it'll be nice to have options

nostr is allowing for more options

and Jack Dorsey happens to have put up a 1 billion sats bounty to """Create a “complete” Nostr-based suite of git tools, such that projects like bitcoin-core are sufficiently confident to move away from GitHub."""

https://bountsr.org/nostr-based-github/

They'll have to catch me first.

That's a term of service that I'm not too worried about. I'm not going to cry if they terminate those two accounts (main account is paid, so that's not in violation). And I'm not doing anything unusual or abusive with the accounts so I'm not worried they'll take action anyway.

This kind of information has become even more easily accessible for everyone with the introduction of the Activity tab. I really wish that force-pushed commits and deleted branches would be permanently lost instead of being permanently stored like on a blockchain.

https://github.blog/changelog/2023-05-31-view-repository-pus...

Since people can't behave and come up with "business models" like this, sites will have to become subscription-only.

Is this even GDPR compliant ?

GitHub is already a GDPR joke. They won’t delete anything less than an entire account

How so? I deleted my previous accound and they replaced all interactions with "ghost". Deleting all interactions would be going a step too far IMHO since it would destroy documentation of issues, etc, just like public forums usually argue against deleting all posted content when you delete your account.

Based on Serbia but operating the servers in Frankfurt. With the cavalier attitude to scraping and linking people's identifiable data without any sort of opt-in I had assumed it would be a US company.

Lol let me introduce you to a little organization called the National Security Agency, with their "creepy" periodic snapshots of much more intriguing datasets.

"Stellar Wind" is a good place to start.

Including, but of course not limited to, every communication made by any person within the United States (or outgoing) for the better part of two decades. Internet traffic, communications, all of it.

https://oig.justice.gov/reports/2015/PSP-09-18-15-vol-III.pd...

Looks like I'm destined for the gulag for sure.

Don't blame the archiver who is only doing what is allowed of them by Github. If you don't want to be public don't be public. Other archivers aren't making themselves publicly known but have the data all the same.

Just because you can perve on people at a nudest beach doesn't mean you should.

Society frequently has norms and unwritten rules that we usually follow. Not every rule needs to be written down as law.

Those beaches have rules. You are not allowed to take pictures.

While on github there is no rule like that.

Nobody is claiming there was any rules being broken.

Doesn't make it any less creepy.

If you put information online publicly, you should be always working under the assumption that it will be immediately archived by someone. Whether that is the Internet Archive for websites, a Discord bot archiving edits and deletions, Pushshift (formerly) for Reddit, or just some private group operating a web scraper.

At least in this situation the archived data is public.

The background with static noise really bothers me. Will have to skip reading till they provide a disable button.

On Firefox and even Microsoft Edge there is a "reader mode" option for most websites. I click on that often enough when I expect an article, to remove noise from ads.

No more noise!

The article leaves a bitter taste of unnecessary complexity. Data engineering should not be hard.

For example, you can load the GitHub Archive to ClickHouse, and it will be accessible with interactive real-time queries: https://ghe.clickhouse.tech/

Was this written by GPT? I was quite interested in the topic of the article but I started to get the brain fog I associate with parsing ChatGPTs convoluted sentences.

It also seems like a thinly veiled piece of product marketing.

This

Agreed... I wanted to understand what it was all about, but really struggled to follow. They talk about the whole thing taking around 24 hours, but some part took over 30. Also that it ran on a 4GB of RAM machine, but they needed larger ones to do all the parsing.

Also in the end, unsure of what the actual results are. Maybe I missed clicking on something.

You can get this data for free on Snowflake: https://app.snowflake.com/marketplace/listing/GZTSZAS2KJ3

Caveat: you have to have Snowflake account

Reply on news.ycombinator.com