rbtying

I found this a couple months ago and it was an amazing resource (thanks dmitrygr)!

I ended up using atc1441's base station firmware[0] because it's a bit of a pain to actually pop open the case and get the programming pins on the Chroma 74

[0]https://github.com/atc1441/E-Paper_Pricetags

It's been a long while since I worked there, but IIRC one of the big issues at the time was that the symlink "support" couldn't actually work properly:

- some users want symlinks to be synced as symlinks (and they might not know it, old Apple file formats have internal symlinks), so their files got corrupted if they were accessed on another computer - some users want the sync client to follow the symlink and copy all the files at the destination, which requires the client to listen to filesystem events for potentially the entire filesystem, which is no longer permitted in many cases

so, not exactly a vulnerability, but a mix of security and product issues.

another former contributor to pb-jelly, though no longer at Dropbox.

protoc plugins have an interesting bootstrapping problem as well: the protoc-gen-$LANG interface requires the ability to ser/de protobuf messages that describe the proto file's AST. If your build system builds almost everything from scratch, including the protoc plugin, this means that you need to have a variant of your protoc plugin linked to a working proto implementation...

That's not to say this is impossible or even difficult, but at the time that I last looked at it (more than a year ago at this point), it made it fairly unpalatable to move the codegen from Python to Rust.

I’ve been playing with Rust on the ESP32 using these tools on and off over the last couple of months or so, relying more in linking to the existing FreeRTOS/C HAL for accessing the peripherals.

One thing I haven’t quite figured out: a substantial number of onboard features are configurable at image build time, which then change the effective C libraries via preprocessor definitions. So, it doesn’t quite work to naively run bindgen on the header file once, and then embed that in a -sys crate and put it on crates.io

I think you can hand craft a -sys crate with all the appropriate config options as cargo features, but that seems both very time-consuming and likely to drift from the newest esp-idf versions. The alternative of running bindgen periodically (e.g. as part of the build process) works fine for one project, but then how do you publish reusable libraries online?

The compiler toolchain can cross compile to esp8266, but I think the pure Rust HAL is less developed there. If you use Rust to write components/code that link against the ESP-IDF, I’d imagine that would work?

I have the toolchain in a Docker container if you want to try it out.

1. We thought about this at one point, IIRC, and various parts of Dropbox sync have been formally modeled. But CanopyCheck and Trinity are useful in part because they test the real, production code -- the hash-indexing bug that Sujay mentioned elsewhere on this page is an implementation error, not a design error.

2. Kind of. There are tests for interactions between instances (in particular, between different Dropbox folders on the same machine), but running Dropbox twice on the same folder is explicitly not supported.

3. Yes, this happens. There's logic to handle these changes, since some applications expect to use in-place edits and others swap in a new file -- you don't actually need grace to have an interesting situation. You can experiment with this yourself in a Dropbox folder :)

I'm not Sujay, but I worked on this system once upon a time too! (not anymore)

1. It does redo the minimization phase, but the actual execution is extremely fast, so this cost is minimal. Storing test outputs gets pretty expensive when you are running millions of tests, and since there are very few failures, recomputing this is worthwhile

2. Yes and no! the article talks about this a little, but the "heirloom" system does essentially this, and the "native" filesystem variant of Trinity runs the same Trinity tester code against a real filesystem. The "no" is due to the issues with randomized testing -- since any operation that you do can affect the RNG, the exact operation that is run for a particular seed can change if you swap any part of the system. For regression tests, the operation sequence can be put into a separate, non-randomized test.

3. both

4. Testing in Nucleus is a sliding scale from "unit-test-like" to "integration-test-like" -- Trinity is mocking plenty of functionality; CanopyCheck is simultaneously testing many different components. It would probably be more accurate to say that CanopyCheck is testing a smaller subset of components with much greater fidelity, and Trinity is testing as much of the sync engine is practical.

I can't speak for Slack, but it's not unreasonable to believe that a single machine's available output bandwidth (~10-40Gbps) can be saturated during a deploy of ~GB to hundreds of machines. Pushing the package to S3 and fetching it back down lets the bandwidth get spread over more machines and over different network paths (e.g. in other data centers)

It depends on what you mean by checking all the boxes:

Go is tricky if you run on diverse platforms and want to do a lot of FFI, since cgo overhead is significant. The type system in Go is also not very powerful, which is both good and bad.

Haskell tends to hit performance walls that are very difficult to debug, and has a pretty similar learning curve to Rust (most people you hire onto the team won't know the language already).

The predominant competitor in this space is probably a high-level dynamic language combined with C/C++ library code. With good tooling and good practices to mitigate footguns, the extensive library support in C++ has a lot to offer.

Of course, I think Rust makes a better trade-off there, but early in the project it was not at all obvious that the good parts outweighed the fact that we would probably have been the biggest user of any library we depended on. We had some fun adventures in stress-testing HTTP/2 support here :)

As I recall, this was investigated at one point on Sync Engine Classic and is definitely a thing that folks were thinking about for Nucleus, for pretty much exactly this reason.

The biggest challenges with this kind of work (not specific to the USN journal) are in the lack of reliable cross-platform support for features which can be shimmed to look similar. The more unique the code path, the harder it is to test en masse.

It's also the case that the execution environment on Windows machines tends to be very diverse due to a long history of backwards compatibility and the ability to set complex domain policies -- Dropbox strives for a good user experience, and "go talk to IT to have them change this setting" is rarely one of those.

disclaimer: worked on sync at Dropbox; don't work on it anymore; don't have current context

disclaimer: was on the team, but didn't write the article / don't work on this anymore

Rust was adopted at Dropbox for some serving infrastructure use cases more than a year before the sync rewrite was started, which was about four years ago. I'd say we solidly predated the "rewrite it in Rust" meme.

I believe that this rewrite was only successful because of Rust's ability to both interact safely/efficiently with underlying OS APIs (they're pretty much all C-like) and to encode complex concepts into the type system and the compiler. Rust isn't the only language with these properties, but it is one of the few -- and it's one that we really enjoyed using.

If you're interested in graphics, animation, simulation, machine learning, or other topics, knowing the concepts in linear algebra will be worthwhile.

Most higher-level math will be easier for you if you have familiarity with vector calculus and linear algebra, so if you're the kind of person who benefits from a structured class when learning new/weird ideas, it's worth studying in school. (I'd also generally recommend a course in probability and statistics)