Hacker News Reader

Comment by zbentley | Hacker News Reader

Wow, GitHub could save a lot of storage space if they dedup'd across projects/files explicitly, rather than storing Git repos, which is what I'm assuming they do.

Even with a good deduping/compressing filesystem, the way git history is stored means that they're probably missing out on a ton of savings here. Eh, it's probably not worth the complexity/deviation from standard Git tooling.

View on HN

Github uses their own storage backend which I believe shares objects across all of projects regardless of whether they are explicit forks or not.

That's really neat. Is there any documentation/discussion available on that technology? It sounds like something that would be fascinating to learn about.

I am not sure if the parent's claim is true, ie that Github is storing objects and sharing them across forked repos.

If they are, then it likely just a direct implementation of git the technology. you can see how git stores data here: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects

Not an expert, but that would only work if there was a single repo, right?

Not an expert either :)

There is the concept of submodules which allows for multiple repos while maintaining the checksum mechanics that allows sharing the same bit of information between branches and across commits: https://git-scm.com/book/en/v2/Git-Tools-Submodules

The trick is that git maintains an abstract file system (ie a graph) across commits. The graph consist of pointers to content without having to create a clone of the actual content for every new version of said graph....it gets a little dizzy to explain but it is really not too complicated :)

Not necessarily. To share blobs, GitHub would need to replace the standard git filesystem blob storage with a distributed database of blobs. All Git repos would share this distributed database.

This talk[1] from git merge has a lot of information about how they run their backend storage and what silly things can happen as a result of that.

[1] https://www.youtube.com/watch?v=-ZNKR9wFe8o

GitHub only shares objects among forks. Source: I used to work on GitHub's Git Infrastructure team, but this is publicly available information.

You can read about their architecture in their discussions of Spokes, which replicates repository networks (the original repository and its forks) across data centers. eg: https://githubengineering.com/stretching-spokes/

Trying to put all of GitHub's object files in a single packfile - even just putting them on a single server - would be impossible.

But even on hosting providers with a bespoke implementation - that do not use core git to manage Git repositories - this would be challenging. We have a custom Git server implementation in Visual Studio Team Services, but it still makes sense to shard object storage with the repository: you have to worry about scalability and performance, but also things like data sovereignty. We can't just put a user's git repository in some global SQL Azure database that contains all the repositories in Visual Studio Team Services: the repositories need to be geographically located with the VSTS account they created.

I vaguely remember that somebody from github claimed in HN comment that all objects are shared.

Puting whole github in single packfile is obviously impractical, but having whole github on some bespoke Venti/IPFS-style content addressable object store is not.

It's not impossible but it too is impractical for performance reasons. Providers that run git core repack repositories regularly because packfiles are efficient; loose objects are horribly inefficient - even on the local filesystem. Moving them to a network filesystem is impractical.

Reply on news.ycombinator.com