Integrating Rust into the Android Open Source Project | Hacker News Reader

Integrating Rust into the Android Open Source Project

security.googleblog.com

92 pointstheafh28 comments

>> Dynamic linkage by default >> By default, the Rust ecosystem assumes that crates will be statically linked into binaries. The usual benefits of dynamic libraries are upgrades (whether for security or functionality) and decreased memory usage. Rust’s lack of a stable binary interface and usage of cross-crate information flow prevents upgrading libraries without upgrading all dependent code. Even when the same crate is used by two different programs on the system, it is unlikely to be provided by the same shared object4 due to the precision with which Rust identifies its crates. This makes Rust binaries more portable but also results in larger disk and memory footprints.

>> This is problematic for Android devices where resources like memory and disk usage must be carefully managed because statically linking all crates into Rust binaries would result in excessive code duplication (especially in the standard library). However, our situation is also different from the standard host environment: we build Android using global decisions about dependencies. This means that nearly every crate is shareable between all users of that crate. Thus, we opt to link crates dynamically by default for device targets. This reduces the overall memory footprint of Rust in Android by allowing crates to be reused across multiple binaries which depend on them.

>> Since this is unusual in the Rust community, not all third-party crates support dynamic compilation. Sometimes we must carry small patches while we work with upstream maintainers to add support.

Having a dynamic linking option for Rust would be a tremendous improvement.

It would help to solve vendoring issues and hopefully improve interop with existing systems and programming languages.

Good dynamic linking support and enhanced interop with C would make it much easier to do a piece-wise replacement of C components with Rust components, allowing for a gradual migration in places where it makes sense to do so.

One good thing about Android in this perspective is that many sub-components of it have their own processes (binary), especially around HAL, after Project Treble [1]. This means that you can rely on the process boundary as the language boundary to start small.

Also note that Android is more C++ than C. And C++-Rust integration is something Rust has already done in Firefox. The road is paved, at least partially. So I have high hope here in long term.

[1] https://android-developers.googleblog.com/2018/05/faster-ado...

> And C++-Rust integration is something Rust has already done in Firefox.

I don't think this is really true.

The simple fact that Rust can't overload functions causes significant grief in a C++/Rust interface, for example.

Personally, I now see C++ as an evolutionary dead end--yes, it's going to continue to exist for a bit until we get a language that is ready for game programmers can use. After that, the fall of C++ will be fast.

This is especially true since we are now back to lots of projects side-stepping the C++ standard libraries. If you're going to do that, why use C++ in the first place?

The simplicity of C is much better suited to being a lingua franca, and much better languages are appearing that can finally be the "Successor to C".

If only,

https://www.khronos.org/sycl/

https://developer.nvidia.com/cuda-zone

https://devblogs.microsoft.com/directx/

https://developer.apple.com/metal/

https://os.mbed.com/

https://www.arduino.cc/

https://www.unrealengine.com/en-US/

https://www.crytek.com/cryengine

https://www.misra.org.uk/Activities/MISRAC/tabid/171/Default...

https://www.snsystems.com/

https://www.codeplay.com/

https://docs.microsoft.com/en-us/windows/apps/winui/

https://www.qt.io/

This is the kind of tooling and ecosystems that Rust still needs to build, besides a plain compiler + cargo.

I wish Google would contribute fixes to the problems instead of working around them. Specifically, they could add Cargo incremental builds, reproducible build mode, and build script sandboxing (run build.rs inside WASM).

That's very exciting. Rust really seems to establish itself as a serious contender for the low-level niche that until so far has almost exclusively been occupied by C.

I wonder how much of an obstacle the lack of the stable ABI and the inability to transparently use C headers still are.

This will be downvoted, but I'm a bit skeptical of the "integrate Rust into big C projects" trend. It seems to me that it adds a lot of complexity: now there are two toolchains you must understand in order to contribute. There will invariably be subtle bugs at the interface between the two languages. Yes, memory and type-safety are wonderful, but they are not the be all and end all.

Does there need to be subtle bugs at the interface if Rust exposes the interface as a C ABI?

I think the idea of this article is that you don’t need to understand two tool chains but instead can focus on the harder task of understanding two languages.

Yes.

Rust does not consume C headers. For every function and data structure of the API surface you gonna need two language projections, C header and Rust source file. For large API surfaces, especially when both sides of the FFI evolve over time, bugs are inevitable.

The problem is not specific to Rust. It affects all languages, except C++ and Objective-C which are backward compatible.

While Rust itself may not directly, tools like bindgen take in those headers and produce the needed glue. https://cxx.rs/ is another example of a tool that's used in these sorts of cases.

You do not have to write these things, nor keep track of changes, by hand.

I wonder why so many Rust FFI wrappers don’t use these tools?

The most popular C library in the world is probably the standard one. Yet in https://github.com/rust-lang/libc I don’t see any glue automatically generated from C headers, everything’s written manually.

Other examples: https://github.com/servo/rust-png/ https://github.com/bozaro/lz4-rs

The libc crate's test does actually parse the C header files to generate tests for conformance.

There are a few big reasons to consider avoiding parsing the C headers directly:

* The C headers might use features in C that are not compatible with the C parser or Rust ABI itself. (C unions used to be a big headache, and bitfields probably still are.). Manually specifying only what you want to import avoids that issue entirely.

* As an extended note to the above, many libraries (particularly your system's implementation of libc) are actually not written in any standard C dialect but instead have a lot of compiler-specific things tacked onto it that make it even more difficult to parse and handle correctly than regular C/C++.

* Parsing the C headers requires having them installed and being able to find them correctly. When you start dealing with cross-compilation toolchains, this can be a major hassle.

The first two are older than these tools, not sure about lz4. Not 100% sure why but I would imagine that’s at least a reason.

> Rust does not consume C headers.

Yet, at least. I hope one day it can, transparently.

The reason I'm optimistic about this trend is that it's been a common thing in game development for a long time (low level language for execution speed, high level scripting language for development speed) and it works really well.

Having to maintain this interface between the language does come with its own set of problems but is well worth the tradeoff especially if you can spend most of your time in one language or the other.

That's not what's happening here (Android already has low level C and high level Java). What's happening here is two low level languages side by side.

Sometimes complexity is worth it for the engineering benefits. We could all be running our software on beautifully simple vintage 1980s microcontrollers: the 6502 was an incredibly elegant and simple design! But then we'd be stuck with computers that can't multiply in hardware or add two floating point numbers or prevent a misbehaving application from taking down the whole system.

Unless one starts with another CFront, naturally moving away from how C compilers see the world has some cost, no matter the language.

You have good reason to be wary, it is added complexity that should not be taken lightly.

The reason to pay this cost is that the alternatives are: (1) do nothing: keep developing solely in C, (2) rewrite the big C project in Rust in its entirety. The cost of option (1) is significant: memory safety errors contribute to 70% of defects not detected by test [1]. Even with static+dynamic checking tools, developers keep introducing bugs into C projects that aren't possible in safe Rust [2]. The cost of option (2) is significant risk. Big C projects have grown over time and have many implicit requirements already satisfied. Discovering all of those incidental bugs and features that are necessary takes a long time and scales with the size of the project.

[1] https://msrc-blog.microsoft.com/2019/07/16/a-proactive-appro...

[2] http://blog.vmsplice.net/2020/08/why-qemu-should-move-from-c...

I don't think the only options are "rewrite in Rust" or "don't". There are other options than Rust.

Not with the same performance and memory safety guarantees.

Prove it. This is going to come as a wild shock to you, but Rust isn't the only language that provides this. There have, in fact, been memory safe, performant languages for decades. But we do live it a bubble here where the norm is anything that hasn't been hyped by a pack of fanbois on HN the last 2 years doesn't exist and "an untruth repeated enough times...".

It’s generally expected that the person making the positive claim demonstrate it. You’re asking them to prove something they don’t think exists.

(I have some ideas of what you may say but am curious! You clearly have strong feelings here.)

That's some serious lazy right there. I'm suppose to demonstrate something they could have learn if they'd done a simple search. Can I google that for you? But that goes back to the whole "if someone hasn't hyped up in my echo chamber, it doesn't exist" mentality that got us here, and there's nothing I'm going to be able to do to fix that.

I would perhaps argue, especially for projects like Android, that memory safety should be considered table stakes going forward. The class of risks that are mitigated or eliminated by memory safety are well worth the effort. There isn't a great excuse to not prioritize memory safety when it's an option.

If the project is a pure C project using a build system that can't integrate Rust very easily, or it would come at a great cost, then I can see this being a totally valid concern.

However, AOSP is massive. It is HUGE. And complicated to build. I really, really doubt this addition will make a huge deal.

This is the same for many of the huge projects adopting Rust. The worst one so far is the Linux kernel IMO, because at least that is mostly C and GNU Make. And yet even there, that still isn't that bad.

Also, as far as bugs at the interface, I'm pretty sure it's a wash. C and Rust are ABI compatible to the extent that matters. The "subtle bugs" are most likely going to be ones that could've existed without the additional complexity of a new language and an interface between them. It's no worse than say, JNI, which doesn't seem to be a serious issue.

I'm pretty sure the benefits of Rust can outweigh this cost. As long as you can build the project, you don't really need to understand 2 toolchains to contribute unless your contribution lands across a boundary between 2 toolchains. On a lot of really large projects this may already be the case anyways.

Please don't downvote-bait. It's tedious and against the site guidelines.

https://news.ycombinator.com/newsguidelines.html

I think it makes the most sense as part of a strategy to incrementally migrate a codebase from C to Rust, or else if there is a clean separation between components such that people are working in one ecosystem or the other but rarely both.

In whichever case, we should be careful not to overstate the complexity of integrating Rust and C. Specifically, bringing in any given C dependency means integrating its bespoke build system (and those of each transitive dependency) with that of your project (or more likely, you punt on reproducible builds and instead push the dependency hell onto every contributor) which doesn't strike me as considerably less work than integrating the Rust toolchain (and since Rust has sane, standard build/dependency tooling, it's pretty much a one-time cost rather than a once-per-dependency cost).

Reply on news.ycombinator.com