Comment by fragmede | Hacker News Reader

Comment by fragmede | Hacker News Reader

Parent Full thread

Git shortcomings are well known by this point, so "all" a successor project has to do is solve those problems. Git scales to Linux kernel sized projects, but it turns out there are bigger, even more complex projects out there, so it doesn't scale to Google-sized organizations. You would want to support centralized and decentralized operation, but be aware of both, so it would support multiple remotes, while making it easier to keep them straight. Is the copy on Github up to date with gitlab, the CI system, and my laptop and my desktop? It would have to handle binaries well, and natively, so I can check-in my 100 MiB jpeg and not stuff things up. You'd want to use it both as a monorepo and as multirepos, by allowing you to checkout just a subtree of the monorepo. Locally, the workflow would need to both support git's complexity, while also being easier to use than git.

Anyway, those are the four things you'd have to hit in order to replace git, as I see them.

If you had such a system, getting people off git wouldn't be the issue - offer git compatibility and if they don't want to use the advanced features, they can just keep using their existing workflow with git. The problem with that though, is that then why use your new system.

Which gets to the point of, how do you make this exist as a global worldwide product? FAANG-sized companies have their own internal tools team to manage source code. Anywhere smaller doesn't have the budget to create such a thing from scratch but

You can't go off and make this product and then sell it to someone because how many companies are gonna go with an unproven new workflow tool that their engineers want? What's the TAM of companies for whom "git's not good enough", and have large enough pocketbooks?

You are right. GIT is not DVFS, its DVCS. It was made to track source code, not binary data. If you are putting binary to DVCS, you are doing something wrong.

But, there are industries that need it, like game industry. So they should use tool that allow that. I heard that Plastic-SCM is pretty decent at it. Never used it so cant tell personally.

Replacing GIT is such a stupid idea. There is no ONE tool to handle all cases. Just use right one for your workflows. I, for example, have a need to version binary files. I know GIT handles them badly, but I really like the tool. Solution? I wrote my own simple DVFS tool for that usecase: dot.exe (138KB)

Its very simple DVFS for personal use, peer to peer syncing (local, TCP, SSH). Data and Metadata are SHA-1 checksumed. Its pretty speedy for my needs :) After weeks of use use I liked it so much, I added pack storage to handle text files and moved all my notes from SVN to DOT :)

Second that way of thinking, for me GIT is as good as it gets for versioning text files.

Not handling binary files is not a downside for me because GIT should not be a tool to handle binary files versioning and we should use something else for that.

What do you use when you have a tiny .png or .jpg that needs to live alongside your source code now?

Just put it in your git repo.

I can put a binary file in GIT repo especially small ones and ones that don't change - things that people want are "handling binary files well", whatever that means, but putting big binaries in GIT or a lot of binary files or versioning them is not the use case for GIT.

I think what they mean by "handling binary files well" is being able to selectively not clone/pull them. Otherwise, binary files are handle as any other files, except you are not shown a diff of them by default.

You are wrong.. Just start putting files of GBs in size and you will see what happens.

So I just added a 2.4GB iso and other than taking longer nothing happened. Push and clone also seem to work.

Right :) I forgot that everyone is in 64bit systems these days. This change a lot. Anyway, GIT uses single blob for single file content. This is all right, if we speak about source code and text files, as those are not large. Thing changes for arbitrary binary files.

AFAIK, not much of anything changes (perhaps name some of these changes?).

And what is wrong with a single blob? This seems work great for deduplication, if you were using git to manage a photo library and happen to add something twice.

There is several issues depending where you run git on. For example mmap() can be problematic on large files in 32bit OS. On Cygwin it gets even worse.

As for deduplication, I do not think so. If you have single blob, lets say 1GB and you change just 1B, whole blob changes and no more dedup. If you use basic method of static block size, lets say 512KB, this will work much better. Futher, there are more advanced techniques to handle dedups like roling checksums to carve out even smaller sub-block.

But you're always arguing the odd case: - 32 bit systems - cygwin git on Windows instead of native - expecting deduplication on different files (???)

Uh, hold on. Are we trying to discuss objectivily? If not, and you just insist like GIT is best for everything fanboyism, and use it everywhere.. I am not interested.

I pointed out that GIT have some rough cases and its good to be aware of them. I said it already and will say it again: choose right tool for the task.

If your workflow with GIT works, use it. No need to discourage other solutions. Yes, I use ancient platforms and I want DVFS on them too.

Well, you said something will happen with git and big files, but it seems not much will happen for the majority of users.

DVCS stands for distributed version control system, it has nothing to do with source code?

Maybe you're confusing it with SCM which are source control managers, that's the only ones that handle strict source only, but scm can mean other things.

Hard to say.. For me DVCS is more advanced version of DVFS. DVCS can do branching and merging, provides more metadata for revisions etc.. DVFS just do pretty much one thing, store binary blobs. And because binary blobs cannot be easly merged, I would not use it for storage here. But I guess, its just me :)

What is this dvfs you talk about? desktop virtual file system? Dynamic voltage and frequency scaling? Google doesn't know, or well, the first page doesnt know.

DVCS stands for distributed version control system, it has nothing to do with binary blobs, whether or not it supports it, is an implementation detail.

VFS = Versioned File System; D means Distributed.

Isn't there git-annex [0] if you want to store large binary files?

[0] https://git-annex.branchable.com/

Yeah I know about git-annex. It might be good solution for big data. In my case, I do NOT want to decouple storage from metadata. I want single repo for single project that is self-contained. Easier to manage, its truly distributed. No need to bother w/ backups because every replica have everything allready. Its good model for several GBs of data.

You say this, but Git has made great strides in scaling to huge repositories in recent years. You can currently do the "checkout just a subtree of the monorepo" just fine, and you can use shallow clones to approximate a centralized system (and most importantly to use less local storage).

> If you had such a system, getting people off git wouldn't be the issue - offer git compatibility and [...]

Git is already doing exactly that.

Git itself isn’t though, not in any real way that matters. Having to know all the sub trees to clone in a mono repo is a usability nonstarter. You need a pseudo filesystem that knows how to pull files on access. And one ideally integrated with the build system to offset the cost of doing remote operations on demand and improve parallelism. Facebook is open sourcing a lot of their work but it’s based on mercurial. Microsoft is bought into git but afaik hasn’t open sourced their supporting git tooling that makes this feasible.

TLDR: the problem is more complex and pretending like “you can checkout a subtree” solves the problem is missing the proverbial forest for the (sub)tree

Microsoft's vfs for git is open source. So is scalar. These are the two main approaches used at Microsoft for large repos. Unfortunately the technically superior vfs approach was a nonstarter on macOS.

> "checkout just a subtree of the monorepo"

How do I check out, eg https://github.com/neovim/neovim/tree/master/scripts into a directory and work with it as if it was a repo unto itself?

You can't (since commits are snapshots of the repo root). You can have this approximation however:

    git clone --filter=blob:none --sparse https://github.com/neovim/neovim
    cd neovim
    git sparse-checkout add scripts

Unfortunately, GitHub does not support --filter=sparse:oid=master:scripts, so blobs will be fetched on demand as you use the repo.

Reply on news.ycombinator.com