672 karma · joined July 13, 2014
See for instance https://docs.rs/sanakirja-core/latest/sanakirja_core/btree/p...
Sanakirja even reached a point where bugs were found only in the last few functions without comments. Writing the comments to explain exactly what each line does in all cases (there are lots of cases everywhere) fixed everything.
However, if you're claiming to make a system including "the best" of these three tools, any reader will expect the obvious property that Jujutsu can faithfully simulate all possible workflows from these tools.
Since it is essentially a layer on top of Git, I am fully convinced that it can simulate Git (for example by using Git directly if needed). But since you fail to explain your algorithms and core ideas in the docs, I'm also convinced that it can't simulate draft changesets from Mercurial.
Since I know what it takes to simulate patch commutation (from Pijul), especially together with the snapshot-based nature of Git, I also doubt that it can simulate that. Obviously, I'd be happy to be proven wrong, because I'd learn something really cool!
I've asked you this question before somewhere else: I still don't know whether Jujutsu satisfies the most basic property any user expects from merges: associativity. If not, it's ok, Git, Mercurial, SVN, Fossil also fail to guarantee it on all merges.
I'm also surprised you aren't claiming to do better than Darcs. Even though I've never used `darcs replace`, and I failed to implement that command in Pijul, I'm nevertheless a bit jealous, because I find that quite elegant.
I'm convinced there is a good theory. Moreover, since line graphs (i.e. totally ordered bytes in a file) are just a particular case of trees, that theory could even be implemented in the same way Pijul is now, where the "easy" case just means blobs linked together.
But I'm also convinced that there are so many edge cases that I wouldn't want to even start considering that before Pijul sees some real-world usage at a decent scale. If everybody misses the point and ends up preferring Git, then why even bother?
> though I don’t see that behaviour with operating systems. Perhaps they are considered too sacred or something.
I guess everybody sees how they would start to write a VCS: just read files, maybe write diff, imagine some datastructures to store that, fix the bugs and problems, that's it. Fewer people can see how to write operating systems, and even people who follow hobby tutorials immediately realise the complexity of handling all the hardware.
> I think most programmers aren’t stressing enough file systems enough to find that they are sometimes buggy and hard to deal with accurately and efficiently, and if pressed might mumble something about calling fsync.
That's right, but on the other hand if you look at the "massively parallel, structured filesystems", called databases, the space is tiny:
- There's a multi-billion-dollars company (Oracle) built at a time where nobody wanted to write fast databases.
- An academic project (Postgres) which grew extraordinarily slowly, and somehow managed to survive as a research platform, funded by public funds.
- Two single-author projects led by extraordinarily motivated people, one of which isn't meant to support concurrency (SQLite), and MySQL/MariaDB, built because there was no affordable, non-experimental database solution.
- Recent years have seen some progress funded by giant companies like Google for their cloud infrastructure, but we're not yet seeing many CRDTs in production.
The same goes for theory-based programming languages: both Haskell and OCaml have barely survived for 20 years before being used at massive scales. Rust has had a slightly easier time, but not much easier, despite being supported by Mozilla.
Yet, most engineers would consider databases "basic" technology, because they use them a lot, and (quite fortunately) don't need to implement them to build on top of them.
- There is no good diff for trees. Some formulations of the problem are NP-complete, meaning that patches won't be minimal, and hence users will get unexpected conflicts.
- Most VCS users have learned to be afraid of their favourite tool, and treat it like something extremely fragile that can break at any time for unknown reasons. Words like "porcelain" show that feeling is even present in the tool's authors themselves. For that reason, the more layers and heuristics one adds on top of basic tools, the scarier it becomes (especially for experienced users).
Writing a VCS is hard and takes way more time than most people expect. Few applications share the stateful nature of VCSs, one needs to feel the pain of designing and implementing databases and/or filesystems in order to understand these systems, where any bug can mean the state becomes corrupt, data is lost, etc. which is the very thing the system you're writing is meant to prevent. So, that kind of effort will only make sense when there are enough users to justify it and support the development, in terms of both man-hours and money.
At the moment, Pijul is in a state where there is just one remaining feature I want to implement before beta, with a massive expected impact on disk space (and a limited impact on speed as well).
For the kind of "smaller (but essential) features" like what you're suggestion, there are tons of those, and we can totally implement them after beta. Git export is a common request we get, though. I'm just scared of how people will react when they export and then reimport their patches elsewhere, and get tons of fake conflicts.
One issue with two-way interop is that while importing a commit into a patch is easy, doing the opposite is more complicated. Indeed, since Pijul is patch-based, there is no way to map patches onto commits in a unique way, since a single patch is quite independent from the context (other patches around it), which is what makes Pijul easy to use.
Also, if Alice and Bob work on a common Pijul server in parallel with their Git server, they'll get artificial conflicts in Pijul, even though they have the same contents.
Then, I don't think you actually understand what we mean by patches: for anyone who has looked into the topic, patches are obviously not equivalent to commits except in the most basic cases (where all tools "work", in your own terms).
A datastructure is defined only with respect to the operations you define on it, I'm sure you use other commands than checkout. For example, people who review snapshots rather than patches before a merge (or even a commit!) either have really small projects, or way too many man-hours to spend.
> if you take one or two hours to learn the documentation is actually intuitive
Git isn't "intuitive" at all for anything but the most basic operations (I agree it is easy for those, if you are willing to work with snapshots). For example, the existence of `git rerere` clearly shows that Git doesn't model conflicts at all. Then, merges are not unique, Git chooses one at random when there are more than one. Rebase is a horrible hack, "replaying" commits doesn't even always work.
Moreover, saying Git is easy isn't a very effective way to show off on HN: if you feel you have to defend Git so strongly, even though it wasn't actually attacked, and to the point of refusing to reply directly to any rational argument, it clearly shows you invested enough time in Git to feel you would lose something if you had to move to a better solution. So, sorry, but I don't think it was just "one or two hours" for you.
Then, Darcs has never been slow for me, I've been using it for about 15 years, I can remember of one minor performance issue. It doesn't scale well to very large projects, but there are extremely few of them. Since concrete arguments are important when discussing with others, the main event which started Darcs' decline was its inability to handle conflicts in GHC (the Haskell compiler), which was at the time the open source project with the largest history (as measured in number of patches).
As for Pijul, I'm curious where you got your information from. I know Pijul's authors well, at least enough to know that they are well aware of the algorithms used in Git. Pijul does solve a number of problems:
- I know at least one very large-scale project where collaborators are well aware that 3-way merge can mess with their code in really bad ways, and complain about wasting tons of time with that. Unfortunately, I can't embarrass the project's authors by sharing their name, but I can say that it is a widely used piece of software with a really long history.
- Knowing that the code you review is the code you merge is the only thing you should care about if you care about your code at all. This is not what 3-way merge gives you, as the Pijul manual clearly explains (see "lack of associativity").
- Patch commutation is what Git tries to do all the time (rebase, merge, rerere…), except with bad hacks which fail in the more complex situations, which is where we need them the most.
- They are not "associative", as mathematicians say, which means that if a remote branch has two commits A, followed by B, merging A then B might give you a different result from merging just B. Stockholm syndrome has convinced many programmers that this isn't a problem, but it actually prevents the very sane reviewing process of reading individual proposed changes along a branch. In other words, this is one of the main reasons Git users have to follow "flows".
- Conflicts! 3-way merge treats conflicts as a failure to merge, and tools based on 3-way merge (Git, SVN, Mercurial, Fossil, CVS…) stop everything when a conflict happens. Now, remember that these tools are based on snapshots, and snapshots can never have conflicts. Therefore, conflict resolutions are forgotten, the only thing remembered is that "Commit [insert SHA1-hash] (Revision number [insert integer] in SVN) comes after these two parents". This is so wrong that Git even has a command to "pseudo-fix" it, called `git rerere`. Again, if this doesn't sound like a problem to you, this could be because Stockholm syndrome taught you that the tool gets upset if you merge from the same remote branch twice, or if you fork a branch that isn't called "master" and isn't on GitHub. In other words, because you're actually using a centralised version control tool ;-)
A bit unfair, Darcs is indeed less fashionable nowadays because of performance issues on very large instances (which few projects actually have), Pijul solves these problems but is still alpha. In other words, this lineage of version control is between two different tools.
That's actually what Pijul does under the hood, its CoW filesystem is called Sanakirja (Sanakirja is a bit more than just a filesystem).
(I'm one of the authors).
The word "partisan" is a bit ambiguous in your comment: just to clarify my position, disagreeing with one side (however strongly!) doesn't mean supporting the others.
The reason I wrote that lengthy argument against choosing Scaleway for this project, is that Scaleway was in a grey area for me: while it is obvious that the other two providers I mentioned aren't a good fit for this particular project, Scaleway is extremely competitive in many regards, and I've had to go through a lengthy reasoning to imagine what could go wrong for this particular project.
That said, I can see how a lengthy reasoning shouldn't always imply a lengthy phrasing, and in fact I agree that the statement in the post seems to mean the opposite of what I meant: since I wrote many more words about Scaleway, it does seem like I was trying to be mean to them, which is quite the opposite of what I really meant.
So, thanks for making me see that, I've fixed it in the post.
> but saying they "organized a militia against its fellow citizens" is about as honest as saying "the US government is running extermination camps" because it detains immigrants.
Detaining migrants is not exterminating them. Private people paid by to go beat up random protesters are usually called militiae. Moreover, Valeurs Actuelles is quite different from Fox News: in particular, Fox News is not explicitly promoting the slavery of black people, nor promoting military coups against the US government.
My only point here is: I believe people should be able to express and discuss any opinion that doesn't contradict my freedom to disagree. Note that in my (now amended) blog post, I didn't mention anything "political" in the sense of "open to debate".
> Scaleway is out of bounds because its largest owner is a supporter of the President of the Republic
Sorry, but you got that completely wrong. I obviously failed to communicate an essential difference I see between what I call politics (i.e. debatable opinions) and the "other thing", which is a denial of the basic right of others to disagree.
My point in the post was, someone who buys press outlets might not hesitate to do the same for contents they dislike on web services they host, or at least there is no strong indication that they won't do that. Given the fact that some cloud providers have done it in other European countries (as stated in the blog post), I wouldn't be surprised myself.
But even more strongly, I don't understand what to expect from that "other thing", and that scares me.
> What's next, you're not allowed to connect to a Pijul database if your ISP is Free
Again completely wrong (ironically, my mobile phone is on Free, and I've never used another ISP at home): the perspective from an ISP point of view is different from the hosting point of view. If one ISP decides to censor a website, this doesn't hurt the website very much, since users can change at a very low cost. This doesn't apply for cloud hosting, where the cost of moving can be really high.
> You're not allowed to contribute to the code if you were taught at the 42 School? (yes, I'm caricaturing, but still)
How is that even remotely connected to the issue of deciding where to host a web service?
> But advertising "we won't work with this company because a shareholder is a support of the president and the president went on French Fox News once" is the kind of ultra-divisive
It is actually the opposite of divisive: I want my service to be usable by people regardless of their political opinions, and this is only compatible with basic values of respect, tolerance and free speech. What I'm talking about here is even weaker than free speech, it's only the right to disagree without being humiliated (caricatured as a slave) or beaten up.
I had understood that, but code I don't write is code I don't have to maintain, so I'm obviously always totally open to suggestions (getting comments and suggestions is part of the reason I wrote that blog post to begin with).
No, and yes. The replica have a very large ping between them, serving everything from the leader would be extremely slow.
> The described setup looks like something that cockroachdb would be able to handle nicely, which perhaps is a little more battle-tested than what you wrote.
Thanks for the suggestion, I wasn't aware of Cockroachdb, I'll try to see how I can switch to that.
Also, "what I wrote" is just a small script to extend my previous setup (which used PostgreSQL) to replicae. I first read about Patroni, supposedly used by large companies, and went with that, before discovering that it requires a lot of monitoring, and can't work for a one-person team.
I'm glad to know that you agree with that distinction.
As a user of both, the price different may not be 75x, but I wouldn't be surprised if it were in the 10x-20x range. The polish is also quite different, and is consistent with the price difference.
That isn't true. In Pijul, either patch1 explicitly depends on patch2, or patch2 explicitly depends on patch1, or else these two things you said are equal.
Patches are much easier to work with, more reliable and fully deterministic. For example, merge and rebase are the same operation in Pijul, you can remove an old patch without changing the identity of all the patches after it, and yet have strong version identifiers, with the exact benefits you describe for snapshots.
You can also do partial clones in Pijul: since patches commute, the patches you produce on top of a partial clone can be pushed to the full repository in the exact same way.
https://pijul.org/manual/why_pijul.html
There are many answers:
1. Commutation makes your life easier because you can be much less careful about how you manage your branches. Rebase, merge and commit are the same operation (apply a patch), without any loss of power: you can simulate all of Git within Pijul, except for bad (i.e. silent non-associative merges), which Pijul doesn't have.
2. Everything is easily reversible. I know all actions in Git are reversible in some way, but not in the same way: for example, you can't undo an old patch without changing the identity of all the patches after it. I know you're thinking this is important for strong version identifiers, but Pijul also has strong version identifiers, just without the compromise on usability. This is achieved using cool cryptography tricks.
3. Solving a conflict in Pijul actually solves it. Conflicts happen between two (or more) patches, and are solved by a patch: if the same two patches are on another branch, you are guaranteed to get the exact same conflict in 100% of cases, and that conflict is solved by the very same patch that solved it in the first branch.
4. When merging, Git solves an optimisation problem that may have multiple solutions in some cases. Git chooses one arbitrary solution based on the content of the lines, and doesn't warn you if there are others (because that would be a very hard computational problem). Pijul doesn't do that, and gives you strong guarantees on merges. You still have to test, but when reviewing, you can predict in your head, with 100% accuracy, how Pijul will merge. This isn't the case in Git: lines inserted at the end of a file might be merged into unrelated lines at the beginning of a file sometimes, if Git feels like it.
1. The order between lines is preserved by Pijul. This is important: let's say Alice works at the beginning of the file (lines 1-10 of 1000), whereas Bob works at the end (lines 990-1000 of 1000). Pijul preserves the order in all cases, whereas Git might randomly decide, based on the contents of the lines, to merge Bob's new lines in the middle of Alice's new lines.
2. Git solves an optimisation problem (3-way merge) that may have multiple solutions. Unfortunately, there is no way to count the number of solutions, or even to tell whether there are multiple solutions, in a reasonable amount of time. Git therefore picks one solution silently, based on the contents of the lines. In contrast to that, Pijul is deterministic.
3. Pijul is associative, meaning that merging A and (BC) is the same as first merging A and B, and then merging C; in other words, you can merge a branch commit by commit. Git doesn't have that property: if you merge a branch, you MUST (1) stop working on it, or else the future merges become totally unpredictable, and artificial conflicts might come back (yes, I know about dirty-hacks-to-try-and-fix-that-when-they-work such as rerere), and (2) check the result of the merge extremely carefully, because in addition to the logical errors that merges can reasonably introduce, Git might also introduce extra unpredictable errors by randomly shuffling lines around.
However, Pijul was inspired by this, with a number of differences, including support for line deletions (which changes everything) and multiple files, which cause many more conflict types.
Darcs was another major source of inspiration.