jamwt

It's using the Rust native arrow library, from the DataFusion project. So nope, pure rust.

https://github.com/apache/arrow/tree/master/rust/arrow

It's... been hyper for years and is still hyper. Reqwest is an ergonomic wrapper on top of hyper if you want a higher-level API with less control.

Among the Rust ecosystem things to gripe about, this is a odd one.

async_std vs. tokio is the real split-investment battleground these days, effectively undermining the async/await momentum right as it gets out of the gate.

Ouch, pretty dismal take on Dropbox.

> they have never been profitable

Technically true, but mostly b/c of stock comp. Free cash flow positive for quite awhile:

https://www.macrotrends.net/stocks/charts/DBX/dropbox/free-c...

Not exactly burning money. $1.2BN in cash holdings and growing, projected for an additional $1BN this year.

> have said in their own filings they don't know when or if they will ever be profitable

.. in 2017, and this is basically just boilerplate. Just about every company that goes public says this in their S-1 these days.

Now, 3 years later: https://www.fool.com/investing/2020/02/22/investors-cheer-dr...

> I like Rust but it really irks me when people pretend it is as easy as C++. It isn't. It is much harder

No it's not.

"hard" is not single dimensional.

"hard" to write your first hello world? No, both are easy.

"harder" to write your first moderately-sized program that basically works? Yes, for almost everyone. You'll fight the borrow checker. The learning curve is real.

"harder" to write your first large/medium, correct program? Definitely not. The safety will save you, here. It's not just memory. It's no nulls, it's no exceptions, pattern matching exhaustion enforcement, baton-passing session type patterns, race-freedom, and so on. The "it compiles, it works" dynamic is akin to OCaml/SML, if you've used those at all. Except it's even better because you get prompt resource finalization.

"harder" to write your nth program after ~6 months of usage? Not by a long shot. Rust is actually an incredibly expressive language. Once you learn to shape code the way the language expects, you'd be shocked at how productive you are. It's in the same ballpark as go, perhaps a little better if you're doing something nontrivial due to the power of generic libraries.

I spend almost no time "fighting" the borrow checker, as do most ramped-up Rust programmers. It's pretty much all win at this point. It's a different category of language than C(++). It actually gets a little exhausting hearing people talk about how hard the borrow checker when folks have barely used the language.

These days, the only time I'll use a GC is if I'm writing something _very_ fast and short in Python. An experiment, a prototype, or a tool. A short REPL session. I really do not miss GCs otherwise.

But yes, compile times are somewhat slow. If you go template crazy in C++, you'll be in the same ballpark. But the "equivalent" C codebase will smoke it.

Sorry, not super familiar with Syncthing. It's definitely not impossible to do better than Dropbox, but to some degree, as in all things, it's a function of engineering resources, telemetry, and usage.

We've likely put 25-100x more resources into solving this problem over the last 10 years, and we just have a lot more data b/c we have 100s of millions of users on lots of platforms using Dropbox with every application you can imagine. So we're able to tease out the "long tail" of weird file system and application behavior in a way that's very difficult for smaller projects. Truly durable conflict management in the face of arbitrary mutations by user applications on the filesystem ends up being a really, really hard problem to cover exhaustively. The Dropbox client handles literally hundreds of special cases.

So, yeah, I believe it is (in general) safe to assume that Dropbox is probably doing A-more-correct-thing for a complicated (and admittedly confusing) reason when it comes to sync behavior. But we're not perfect--we do still find surprises from time to time, so feel free to contact support if you see something that looks wrong!

Just a heads up: that almost certainly means SyncThing is failing to recognize some classes of conflicts and concurrency races, and is destroying some changes. Doing this right is a hard problem. I know they can be a pain to deal with, but we don't spin out conflicts for no reason!

Disclosure: I work on the Dropbox sync engine.

Yes, hate, hate NDEBUG. Such a terrible idea to disable assertions.

I didn't say it was! I said it was an "excellent choice". Right?

Yep, that's what I meant about "large". If you have a large project, with many developers you're going to want types + generics to maintain it over the many-years and 100,000s to millions of SLOC life.

I'm not saying unproductive. I'm saying, we need to be realistic. Compared to, say, Python, it's nowhere close.

Compared to C++, Java, or even Go, it's anywhere from competitive to significantly better.

If your primary goal is productivity, I don't think rust will ever be the right language for you.

If you want to remain relatively productive while building things that are one or more of (large, fast, safe), it's an excellent choice.

Hmm... you guys sure have pretty thoroughly sanitized all references to Stud at this point in code, documentation, READMEs, etc.

:-(

Rust:

https://gist.github.com/jamwt/60f66f0fd27d349fe460051e55e79a...

(Also submitted to blog author.)

> Things like placeholders (infinite)

Yep, we're definitely aware of MS's placeholders. Infinite is different in several key ways, not the least of which is that it is cross-platform. And doing this well on Mac is particularly tricky.

More detail on the tech blog:

https://blogs.dropbox.com/tech/2016/05/going-deeper-with-pro...

I'm a sync quality engineer at Dropbox--on the team that writes and maintains the sync engine. So caveat, I'm clearly quite biased. :-)

> They don't do anything that their competitors (OneDrive, Box, Google Drive, etc.) don't do better.

I'm not sure on what you base this opinion. We do sync much, much better, and that's no coincidence--we work very hard on it (I'm new on this team, so I deserve essentially zero personal credit). It is our business. We're not an advertising company--doing sync well translates directly to dollars for us.

On a technical level, it is extremely nontrivial to get right multiplatform, in the face of races, vague filesystem semantics, application-specific behavior, offline, etc. Our competitors aren't even trying things like infinite (https://blogs.dropbox.com/business/2016/04/announcing-projec...).

We're not sitting still, though. We're actually building a next generation sync engine right now (btw, more Rust @ Dropbox). The plan is that this will put even more distance between us and others.

> The "software majors" do not.

Not really true. Most or all of them do for the right candidate. They're not super vocal about it, though.

Purely OPVault!

We've got pretty good OPVault support going, but your guys' RPC protocol is not open, right? Between mini and the browser plugins?

Coincidentally, a bunch of Dropboxers are working on a version of 1Password for Linux written in Rust (it's Dropbox's "Hack Week" right now). Hopefully we can open source it soon. Looking good so far!

> Would you be able to give more details as to how you use disks as raw block devices, bypassing the filesystem? For example, the kind of abstraction layers you use, their interfaces and methods, disk layout of extents, metadata layout, consistency issues, fsync issues, and the various performance advantages from bypassing the filesystem?

Yeah, we're going to write a whole blog post on this sort of stuff sometime soon. Needless to say, it's a big topic!

Thanks!

We're expanding our use of Rust into even more bold endeavours, details soon.

Based on our experience with it in the last 1.5 years, stick with it, it will return your effort many times over in the medium-long run. Then again, if there is no medium-long run for your project (as an entity needing maintenance), just use Python.

Python 2 isn't going anywhere so long as the multi-million line 2.X codebases at big tech companies exist. Every big Python codebase I'm aware of is 2.x.

Nope, honestly. Go is dropbox's primary infrastructure programming language. There's an extensive internal library/culture/toolset around go, so it made the most sense to use that.

These systems are of such a scale that the materials/power/real estate costs dwarf the engineering people costs.

The tradeoff, it depends on your business and your team, your risk tolerance, time scales. But a rule of thumb might be if you don't have more than 10PB or so, you probably shouldn't bother.

(We actually only rebuilt two components in Rust--most of Magic Pocket is still go.)

Memory usage/control, easy FFI with C, safety/correctness of code. Those were the big advantages for our projects.

Edit: deterministic destruction of resources is pretty great too. Yay RAII.

Edit 2: So is the ability to use the "standard" C tool suite for performance, memory, debugging, etc. valgrind, jemalloc profiling, gdb/lldb, perf. All that stuff.

We're going to go into detail on that (the OSD component) in a future blog post... but short version, yep, it's a custom "filesystem" directly done on the block device.

Correct, they're all 64-bit linux. We might use some Rust on 32-bit soon though. So this is interesting to us.

We actually use several different servers from different vendors, and some of them are more customized than others.

Live writes are written out with 4x redundancy in the local zone, then asynchronously replicated out to the remote zone. Some time later, it is erasure coded into a more space-efficient format, independently in each zone.

We use a custom version of jemalloc, with profiling enabled so that we can use it.

We also tweak jemalloc pretty heavily for our workload.

Yep, vendors do sometimes provide stock to large customers ahead of general availability.