Apple is starting to use Rust for low-level programming
twitter.com
twitter.com
Every large company I've worked at has had many, many internal services using all sorts of different languages/frameworks.
This makes more sense for tech companies. I work mostly for large banks and there definitely is less experimentation but there still is more experimentation than one would think (I have couple large Clojure projects very close in my org).
If the organization did not believe in potential of value from these ventures there would just be a small limited amount of languages to choose from because it is more economical when you consider support system. This is starkly visible in banks where there is huge need for development support system but no belief in value from ventures into obscure languages. The end result being that everything is written in Java and maybe couple other JVM languages (just because it is easy when you are already invested in JVM).
if the team doesnt like rust, they probably wouldnt use it less they are forced.
However people with less experience often assume those big companies have strategic directions for tech stacks, and that one teaming using something must be really important and implies bigger usage. I might even have assumed the same when being fresh out of university.
Therefore any news about "X is using Rust" seem to get a lot of traction.
Corollary is that the greater the fraction of reports of Rust usage that we see are about somebody in a big company, the lower the actual usage is.
Similarly, the greater the fraction of articles about Rust usage that carry on about how great it is, vs articles complaining about it, the less actual usage there is. Engineers are always limited by our tools, and we deeply resent those limitations. People complain a great deal about C++; the more useful it has turned out to be, the more complaints they have about it, and they are generally not shy about telling you them.
> We interface directly to low-level Linux kernel interfaces, using asynchronous I/O and threads to distribute workload. Following a very successful first foray into Rust we are migrating an established codebase from C to Rust, and building new functionality primarily in Rust.
However, the team specifically works on the cloud services:
> We develop and deploy software which forms the foundation for some of Apple's most important services, including iCloud, Maps, iTunes, and more.
So no rust-based software is going into MacOS or iOS quite yet.
0: https://jobs.apple.com/en-us/details/200170723/software-engi...
And I was thinking why did they make Swift for low level usage if they were going to use Rust.
Swift is great for application programming, but for low-level performance-critical systems programming it’s currently a huge pain in the rear. Swift’s philosophy is basically “give everything a Python-like level of dynamism, and then heavily inline/optimize out when statically possible.” For instance:
- Swift’s memory management is based on reference counting. It’s usually convenient and efficient (and lets you do cool things like copy-on-write), but when you know more about your object lifetimes than the compiler, the unnecessary retain/release traffic gets really annoying and is not easy to avoid.
- Swift’s generics are dynamic by default, which involves wrapping up every object into a possibly-heap-allocated container and using dynamic dispatch for everything. Usually the compiler will inline/specialize generics, but there’s no guarantee, especially when calling generics across module boundaries (due to ABI stability restrictions). Rust/C++ exclusively uses compile-time template instantiation, which is a little less flexible and doesn’t guarantee ABI stability but it always guarantees unimpaired performance. See [0] for a more detailed comparison.
- No stack-allocated arrays in Swift; all collections are reference-counted and stored on the heap.
- Still no async/await (or any real concurrency model at all) in Swift; it’s planned to be available in the next year or two. This could be a big deal for a networking team.
[0]: https://thume.ca/2019/07/14/a-tour-of-metaprogramming-models...
You're right about the dynamic dispatch, but both specialized and unspecialized generics use the same memory layout. While unspecialized generics have to compute the layout dynamically if needed, there's no boxing required. If this were not the case, specialized and unspecialized code would not be able to interoperate (and as you said, this is a requirement because of ABI boundaries).
> - No stack-allocated arrays in Swift; all collections are reference-counted and stored on the heap.
There's a compiler optimization pass which stack-allocates reference counted objects with a known lifetime (which includes the backing storage for arrays). However, just as with generic specialization, this is a best-effort optimization and is not guaranteed to actually take place.
That said, as has been pointed out, this seems to be one team making a technical decision for their own work.
IMO, having one dominant server side language (with of course some small bits written in other languages) is about as common at a large tech company as being truly polyglot, where you can’t point to a single language used in the vast majority of the BE.
Nobody ever accused us of going all in on Flash.
Especially the ruby and javascript communities are very popularity-obsessed.
I also think we should understand someone was probably quite excited to see something they liked taken seriously by a company they respect. Maybe in their excitement and rush they user hyperbole.
I was in a Perl shop many years ago and introducing Python was strongly discouraged.
You might be surprised. At most companies I've worked at (over a dozen) that's largely true. Three jobs ago I was product architect for the entire company and spent a significant part of my time fighting just this kind of language proliferation. However, it's a very different story at a large company. Google seems to try pretty hard to keep this under control. OTOH, Facebook (where I work) prides itself on a "bottom up" "engineer driven" culture in which any such top-down prescriptivism doesn't fly. Don't know where the others fit in along that spectrum. The situation gets even murkier when code is inherited via acquisition or imported from open source. So such code often exists, whether it's "supposed to" or not.
Therefore in practice, giving engineers the freedom to make their choices doesn't lead to an explosion in the number of languages used in production.
An engineering team that can get started with a new, esoteric language without having to first build internal libraries is much more likely to try that new language.
Like Scala or Kotlin for example?
This requirement is a stumbling block for many languages. For example when compiling Rust, no one actually calls rustc directly, everyone uses cargo. Effectively this meant someone had to reimplement cargo within buck.
When you think about the dizzying array of touch points (logging, metrics/monitoring, service publishing and discovery, and on and on) that even the smallest component needs, the absurd wealth of libraries better than anything in open source, the standards-committee level expertise an email away that you have in one of the mainstream languages it becomes clear that you’d need more than a large incremental improvement in the language itself to break even.
The thing that might get Rust over the threshold isn’t memory safety (modern C++ with hardcore warnings-as-errors static analysis and automated ASAN/TSAN/UBSAN runs is rather solid), it’s build times.
Build times are the thing that will kill C++ unless they can come up with a modules solution that is cheaper to migrate than rewriting everything. At some point you just can’t fucking build the thing no matter how many boxes you throw at it. Rust may have a window of opportunity before C++ finds an answer on that.
Thrift generators and/or a decent FFI can resolve a lot of those. Yes, other teams might try to make noise and threaten not to support you, but I've yet to see one go so far as to blacklist a service because they disagreed with its implementation choices (including language).
> libraries better than anything in open source
Only in their authors' minds, in most cases. NIH at its worst.
> Build times are the thing that will kill C++
Certainly up there, and I wish more people realized this. Every test/deploy pipeline is an order of magnitude slower than it should be because of this, and perpetuating it is part of why I think those libraries aren't really better. In fact many of them make it worse, because they're the most aggressive users of features that increase compilation time.
folly/absl/gtest/glog/tf/pytorch/etc/etc aren’t perfect, but they have stuff that no one else has, and they had it before Github did.
“The guy who wrote the book” so to speak often works at one of these places, and while some of the stuff makes it into open source some of the time, some of it doesn’t.
Google gets Borg, OSS gets k8s.
Citing NIH is kinda flamey as an alternative to citing examples IMHO.
After eight years working solely on open source at Red Hat, I've seen nothing particularly exceptional about the quality of Facebook's proprietary code. People who have no experience beyond school see something new, they think it's the bees' knees, surely they think they must be blessed to work at a place where such things are created ... but they're not uniquely blessed. There are people all over this industry pushing the state of the art. Facebook engineers think that Facebook hires the best people in the world, but people at Google think that they do, and people at Apple think that they do, and people at random startups think that they do. Not without reason in any case, but still incorrectly. It's a common kind of sampling error. The reality is that there's good and bad code everywhere, and the larger a company is the more it tends to regress toward the mean.
I don’t know exactly why a tangent about having F14FastMap or MPMCQueue (and proprietary friends) before the rest of the world has evoked such a strong reaction in a point about C++ making sense at MegaCorp du jour, but calling people who think the cutting-edge internal stuff at the elite companies is a plus for C++ new grads is where I step off this train. I share your frustration with the consolidation of power, money, and talent at a small number of elite institutions to some degree, but you can take it out on someone else.
Rust can at least in principle do better than “recursive textual substitution of massive amounts of redundant code into translation units” and I imagine the Rust leadership is probably sharp enough to know that this is the chink in the armor of an otherwise rapidly improving safety and usability story for a deeply entrenched incumbent.
The rust compiler / internals section has very few experts. The compiler produces poor LLVM IR and compile times are as poor as C++ __Despite having modules__. Basic performance improvements like ```TrustedLen``` took a lot of time to surface despite everyone touting "zero cost abstractions".
Given the hype rust is getting by ruby/js hipsters learning it, you would think they would have a better, faster compiler backend instead of depending on LLVM cruft which can't even utilize rust's Mutable Xor Shared guarantees.
Lumping Apple together with software companies illustrates well why using this acronym out of context is silly. Apple is incredibly silo'ed in comparison. Teams have a lot more freedom since they don't need to reuse internal infrastructure so much.
However once you start getting into very heavily used production services, choosing a different platform becomes increasingly difficult.
Very exciting to see a language tuned toward reliability taking off like this.
A Rust + Swift, massively cross-compiled future sounds pretty nice actually.
I work in arm32 daily, it works well for us. Happy to help if you need some!
https://doc.rust-lang.org/nightly/rustc/platform-support.htm...
https://github.com/mmastrac/stylus/blob/master/.github/workf...
no_std is the result of that. The panic/backtrace machinery turns out to be remarkably expensive. formatting machinery turns out to be expensive. etc.
The fundamental problem is that Rust relies REALLY hard on LLVM optimizing stuff away after code generation. That's not a good way to deal with embedded. It has issues even on Desktop, too (for example, the difference in speed between debug and release modes is enormous--often to the point that if you have performance code you can't realistically run a debug build to actually ... you know ... debug).
There are changes afoot because of MIR, but they will take a while to land.
People forget how many companies created weird and wacky compilers for C on 8-bit targets because C code was "too large/too slow/etc." And how many years those companies persisted.
Optimization for embedded takes time, and the amount of resources available for it is orders of magnitude smaller than general development.
[1]: https://blog.rust-lang.org/2020/09/10/survey-launch.html
Nowhere in the job posting did it indicate or intimate that. It looks like one team doing network infrastructure (for backend services, not Apple platforms) is writing Rust code.
Additionally, having experience with static type systems, the statically typed languages that are transpiled to JS shouldn't be as big a learning curve for you.
However, for a city of 6 million I'd expect a couple hundred C++ jobs.
Saying that they're still using it might be a bit optimistic.
And, there's been some interest towards getting gcc to be able to compile Rust too, so that may happen regardless of its need or not.
Rust is a fringe language and it won't get a lot of traction, because of more-noise-than-signal more-politics-than-technical-talent surrounding it.