briantkelley

> Rust would not have failed to compile at step one necessarily.

True. In this case there was old code and new code. I assumed in a hypothetical hybrid Rust codebase, bindgen would be used to bring the types in old code to Rust, so the compiler would identify the type mismatch in perhaps the impl From.

> If we need to clean up the build and dependency management system to start using Rust correctly, isn't that two major migrations?

At the scale of Office, managing the build system is an evergreen project staffed by dozens of engineers. Adding support for Rust integration is a typical deliverable for such an org.

> My point was that data can be corrupted due to logic bugs as well

For sure. But in the 10+ years I worked on Office, I spent far more time per UB bug than logic bugs. 1 UB bug could take weeks to fix. I don’t recall any logic bug taking more than a few days.

Developer productivity would be much improved if logic bugs were the only class of bugs.

> it's worth thinking about whether it's worth it.

Totally agree we need to evaluate the trade offs of various paradigms. But, I think the most important aspect to understand is the impact on maintenance as opposed to writing.

I’m not sure what my ratios of time spent on reading vs. writing vs. debugging code are, but I am certain writing is what I do least, which is why I’d like to see more research on total cost of ownership.

IIRC, that specific bug was due to a few issues:

1. Mismatched type definitions in different translation units caused an implicit 32 to 16 bit conversion. 2. An addition to the 16 bit value overflowed, causing it to become negative. (The “run for 10 days” thing helped the value get to the point of overflowing.) 3. The negative value led to an out of bounds write, corrupting a key bookkeeping data structure.

Rust would have failed to compile at step 1, and would have panicked at 3 (and potentially 2, depending on compilation settings). I’m not sure what other safe languages are candidates for this domain, but I would suspect these issues would likely be similarly identified.

That bug, unfortunately was identified after release. The release blocking heap corruption bug I mentioned was due to a C++ object being deleted when it called back to an event handler and wrote to its member variables after the handler returned. Ownership and lifetimes would have prevented this design error (which is surprisingly not that uncommon).

> Users don't care if you wrote it in Rust

Totally agree! But they do care if Word corrupts a document that’s been open for 10 days (a bug that a principal engineer spent many, many weeks hunting down).

> If you take 20% longer to write it in Rust, that's short term cost for long term gain, but users don't think that way.

There are many steps between writing code and delivering a product to users. Sure, I can commit some C++ in less wall time than Rust (though I’m not actually sure that’s true!). But when we have to delay a major release by 2 weeks because there’s a heap corruption bug in the flagship feature, everyone loses.

> the pressure is always on

Sigh, yeah. It’s difficult for organizations to act in their own best interests. If a codebase primarily has architecture debt and little implementation quality debt, it’s much easier to keep taking the shortcuts to ship now. But with implementation quality debt, UB is lurking behind every corner and can cause the project to go off the rails at any time. This is why I would like to see more research about the value of Rust with respect to the predominant cost of engineering in extremely large systems: maintenance.

> Users don't care if you need 30% fewer CI servers

Sure, but DevOps does. When we have to drop jobs due to capacity constraints but mark the commit as stable anyway, it’s just a matter of time until a major regression sneaks in.

> The CEO doesn't care if one additional dev has to spend his time making sure asan, ubsan, etc are running properly

Haha, yeah, in practice it‘s always one dev maintaining the asan, ubsan, etc. loops! But, it’s up to engineers writing code to make sure it’s exercised by those systems, so in reality the loops are highly under utilized and eventually hard-to-find UB bug slips in.

> Meanwhile the devs are wrangling the complexity of the software itself, which requires mental resources.

Yes. And encoding ownership and lifetimes into the type system is a huge reduction in the mental tax when working through code that hasn’t been touched in years, or wiring up some feature to an interface another team just landed that’s not documented at all. Not having to reverse engineer the code to discern ownership and concurrency patterns is a huge time saver.

> Any additional complexity introduced by Rust slows down development and frustrates the team

I agree artificial complexity due to limitations of Rust’s soundness checks are frustrating, but a cost I’m willing to pay given the long term maintenance benefits. But the cost of system design complexity has to be paid at some point, and I’d rather pay as much as possible at compile time vs. discovering problems at run time.

> And the bugs are still there.

Certainly. But essentially eliminating classes of UB bugs that are hard to repro saves all the engineering teams time.

> Operating efficiency becomes important only after the product is out and stable, with paying customers and product-market fit

Yes. Total cost of ownership is what I want to see more research in.

And I think this is the source of our deferring perspectives. My original comment wanted more research on TCO, but I don’t think that’s particularly applicable before product-market fit.

> but by then, why boil the ocean with a rewrite?

Who said anything about a rewrite? Use Rust for new development. And, as the system grows, components will be rewritten (whether due to new business needs or to improve maintainability). Don’t keep playing with fire when it’s no longer necessary!

> Sometimes leadership can be sold on a new and improved v2

Perhaps I’m jaded, but my motto is “Rewrites always fail.”

> we need languages that make it less mentally taxing for developers to write product software

For extremely large systems whose codebases have been written over decades, we need languages that make it less mentally taxing for developers to maintain product software. The complexity of maintaining these systems is why FAANG employs tens of thousands of engineers who seem to move slower than an aircraft carrier. The difficulty in reasoning about system behavior greatly impedes progress.

I’m curious why we see so many papers and discussions about the difficulty in writing Rust relative to other languages. I would like to see more analyses of total cost of ownership.

Yes, Rust requires approaching writing code somewhat differently than most popular languages (different isn’t inherently bad), and it’s true borrow checker limitations cause it to reject code that is sound. But that increase in cost at the front end, in my experience, is significantly offset by the decrease in maintenance costs.

I’ve spent a non-trivial portion of my career hunting down and fixing memory corruption issues, race conditions, and other undefined behavior in extremely large codebases. Yes, we’ve seen great advances in tooling in Clang+LLVM, but writing that buggy C/C++ code is still quite easy (a point I didn’t see in the paper). Using validation tools to their full extent is costly from maintenance, configuration, and compute perspectives, and their correctness assessments are less strong than what an advanced type system provides.

Preventing such classes of bugs through an advanced type system seems like a better approach to reducing total cost of ownership because:

1. Some classes of (expensive to fix) bugs are eliminated by codifying system behavior into the type system. 2. The need for a variety of CI runs with Address Sanitizer, Thread Sanitizer, Undefined Behavior sanitizer, etc. and their associated corpus of coverage tests is greatly reduced because such issues are defined away by the language. This saves on CI infra costs and requires engineers to spend less time writing/maintaining tests to exercise the tools. 3. More engineers will spend more time moving the product forward and less time hunting down hard-to-reproduce bugs.

I totally agree with the author’s summary:

> That trepidation and sinking feeling of “What are they going to break this time?” every time the WWDC’s date approaches, has been wearing me down in the past few years. What one should feel, instead, is: “Apple got this. I’m in good hands. I see no reason not to upgrade straight away.” And I haven’t felt this in a long time.

I haven’t looked forward to a release since Snow Leopard, and High Sierra was the only usable version after that. Mojave and onward have been just dreadful. Mac software hasn’t been fun, exciting, or good in a long time, sadly.

I worked on Word for years. Office has thousands of files over 10,000 lines with, uh, various degrees of test coverage and comprehensibility. After some time and experience, your mental model of the architecture ends up being way more important than simple metrics on source code organization.

IMO, organizing source code in files seems archaic. E.g. tracing the history of a function moved across files can be tedious even with the best tools. I’d like to see more discussion around different types of source storage abstraction.

There are benefits of large source files... When compiling hundreds of thousands of files (like Office), the overhead of spawning a compiler process, re-parsing or deserializing headers, and orchestrating the build is non-trivial. Splitting large files into smaller ones could add hours of just overhead to the full build time.