Hacker News Reader

Comment by bluGill | Hacker News Reader

I wish Rust, C++, Go, D... could get together and agree on a common ABI/module format. It doesn't need to be complete (which is to say I'm fine with having to use , just enough that 90% of my program can be written/rewritten in whatever language makes sense and used in the other without having to drop to C.

For starters it needs to have classes (probably using PIMPL like things internally by default). It needs to have some sort of error handling. It needs to support some basic data like std::vector (but they can start from scratch).

Edit: my fingers typed API not ABI first...

View on HN

What you're asking for is a common ABI. To have shared classes, basic data structures, or even error handling, a binary format needs to be defined and adhered to. Of course, to bake all that into languages (and their runtimes), is a great responsibility; but more than that, it can easily be a severe limitation in the way any language may like to do things.

So we have things a little more basic than that, at various levels:

1. API: C. This takes care of a bit more than just data interop and common functions; it maintains an interface, takes care of platform portability concerns (calling conventions, linkage, runtimes), and allows future growth.

2. Modules: From executables and shared objects to services running on a remote machines accessible over TCP/TLS/HTTP(S).

3. Conventions: From free-form (de)serialization formats like JSON/YAML, to read-optimised formats that come with their own implementation code like Cap'n'Proto and Flatbuffers, to share-optimised formats that come with their own read/write/live-share implementation code like Apache Arrow. These come the closest to being a common "ABI", but (thankfully) restrict themselves from being ingrained into a language.

> What you're asking for is a common ABI. To have shared classes, basic data structures, or even error handling, a binary format needs to be defined and adhered to. Of course, to bake all that into languages (and their runtimes), is a great responsibility; but more than that, it can easily be a severe limitation in the way any language may like to do things.

Being able to speak a common ABI doesn't necessarily guarantee that your local ABI has to conform identically to it. In Rust, for example, you need to add a repr(C) to be able to actually guarantee that the layout conforms to the C ABI. CORBA IDL is something that tried to achieve this kind of capability.

> Being able to speak a common ABI doesn't necessarily guarantee that your local ABI has to conform identically to it.

No, but you still need to support the common ABI. E.g., for Rust, that's an easy job, compared to Go.

> For starters it needs to have classes

But, Rust doesn't have classes.

> It needs to have some sort of error handling

But, Rust doesn't use exceptions, just algebraic data types (`Result`).

> It needs to support some basic data like std::vector

But, these aren't primitives, these are just std library constructs. Which is arguably an advantage, if you get a good FFI you can just use the other language's libraries directly.

A higher level FFI is a good idea, I think you're going to discover that the intersection of languages is surprisingly close to C though.

Rust has things that at a binary level are the same as classes, so they could easily be part of a cross-language ABI.

And so what if std::vector isn't primative. It's an extremely common construct that should be part of a cross-language ABI. Hell, it's simpler than strings and they are part of the C ABI.

This is one of those things that everyone is going to nay-say until someone actually does it. The same happened with webassembly. I remember when it was deemed impossible and PNaCl would never work etc. etc.

> Rust has things that at a binary level are the same as classes, so they could easily be part of a cross-language ABI.

They are not "at a binary level the same as classes" (whose classes? C++'s classes?), at least not the moment things like vtables get involved.

Presumably this would be classes in whatever sense the ABI would define them. Which could be defined in such a way that they can be mapped straightforwardly to Rust structs and traits.

> Presumably this would be classes in whatever sense the ABI would define them. Which could be defined in such a way that they can be mapped straightforwardly to Rust structs and traits.

Which would then not map straightforwardly to C++ classes, because the vtable pointer would be in the wrong spot.

You can't have an ABI that makes two things that represent things in concretely different ways somehow resolve that inherent contradiction.

> For starters it needs to have classes

Rust doesn't have classes. And as discussed in the article, Rust puts the vtable pointer on the reference while C++ puts it on the pointee, so even fully abstract C++ classes ("interfaces") and Rust traits (when used with type erasure rather than with generics) are implementation-wise different even if they look conceptually similar.

Go doesn't set up the stack in the same way as the others.

That could actually somewhat work for C++ and Rust (although getting things like generics/templates across would be a huge challenge and you'd lose Rust's safety guarantees at the unsafe C++ interface. Also good luck dealing with exceptions). For Go and D you have a bigger problem: garbage collection. Having two different runtimes play well with each other is far from trivial and could hurt performance.

The main reason the C ABI is de-facto the lingua franca for language bindings is because it's almost runtime-less. You basically only need to agree on things like stack layout and where the parameters/return values go. It's a super low bar.

Dealing with C++ objects, overloaded functions, namespacing, exceptions etc... Now that's a whole different can of worms. How would you automagically map something like std::cout and its operator<< in Rust or Go for instance?

> Dealing with C++ objects, overloaded functions, namespacing, exceptions etc... Now that's a whole different can of worms. How would you automagically map something like std::cout and its operator<< in Rust or Go for instance?

You'd define std::cout and operator<< in terms of a sane underlying reader/writer ABI?

Right, but my point is that then you're generating a complex FFI (which requires high level info about how "operator<<" on std::cout implements streaming and not bitshifting as usual), not a simple binding. Doing that automatically and "standardly" would be quite complex. Those aren't primitive types, they're abstraction built on top of the language's typesystem, g++ is not aware of what a "stream" is in C++, nor is rustc aware of what the Read and Write traits mean.

This has somewhat been solved by cgo by having it documented clearly, and turning a runtime flag on that tells you whether you're doing something you're not supposed to.

This was the dream of C#, Managed C++, F# and VB.Net on the .Net runtime. It worked out OK for some people, but IMO not all that great -- all the languages trended towards being syntactic sugar on top of C#. The hard part is rarely the language, but learning the semantics of the platform the language runs on, and those languages have very different opinions of what that platform should be.

That carries on as UWP, and as someone else pointed to me, Microsoft is going multiple platform with another iteration of it, xplat.

The ABI isn't the hard part. You need to figure out how to track memory ownership when those languages have radically different memory ownership models and often limited support for "foreign"/"unowned" memory.

Passing basic datastructures back and forth isn't hard and there are tools that will autogenerate the code to do that. But when you start using them you'll find they're not solving the hard part of the problem.

No, the ABI is actually the hard part. C++ doesn't even have a stable ABI within just its own ecosystem, ignoring interop with other languages.

Memory ownership is trivial by comparison, particularly in this case when all the languages have the equivalent of malloc/free, so you just make that part of the ABI surface.

On Windows it is called COM/UWP.

That may be portable across languages and runtimes, but it comes at the cost of portability across platforms.

You cannot have it all.

Either you get a bytecode format, or a platform specific one.

GObject (-introspection).

QObject (-introspection).

Interesting enough, the Vala language compiles to C with GObject's for portability:

https://en.wikipedia.org/wiki/Vala_(programming_language)

It was a clever decision. I'm not sure if the right one but a nice thing to try to see how well it works. I wonder if anyone did a "Looking Back"-style write-up on how it's worked out so far with pro's and con's.

The GTK+ and GStreamer bindings for Rust are already mostly generated code (and seem to work very well): https://github.com/gtk-rs/gir/blob/master/README.md

I'm not sure what the exact status of extendin GObject hierarchies from Rust is. See https://github.com/gtk-rs/gobject-subclass

It works relatively well.

I just have two complaints.

The build times, as Gtk-rs seems to do code generation during the build, and the usual Rc<RefCell<item>> for accessing internal widget data on the callbacks.

Vala is.. good and bad. (Like most things.)

Certainly successful at bringing new developers into the Gtk world (especially thanks to elementary OS and its developer documentation). Familiar syntax for C# devs.

The downside is, well, memory bugs. For example, sometimes perfectly reasonable Vala code results in error messages like "Unable to parse accelerator '\u0008\x8dn\u000b\u0008': ignored request to install 501 accelerators" or downright segfaults: https://gitlab.gnome.org/GNOME/vala/issues/626

There's nothing about COM or WinRT that is inherently unportable across platforms. It's just a standard ABI, that can be expressed in C at that - so any platform that has a C ABI can have either of those implemented. It's just that never really bothered.

GObject is a good example of a truly cross-platform ABI like that.

I think Microsoft might also be working on a newer system with that goal: https://github.com/Microsoft/xlang

I haven't really looked at that in detail, but I would find any tooling for better language interoperability very worthwhile.

Many thanks for the heads up, I wasn't aware of it.

Looks interesting, maybe another step into Microsoft Linux.

> I wish Rust, C++, Go, D... could get together and agree on a common ABI/module format.

Don't forget C, with the amount of C code out there you'll have to make sure it can handle this new format. And by the time you limit yourself to the lowest common denominator between all those languages you'd probably just reinvent the C ABI anyway.

The point is the C lowest common denominator is a bit low. I'd like something that by dropping C we can be a little higher.

I'm fine with the ABI having an explicate this pointer (which python already does for classes) which allows C to call/implement classes as well, but it needs to be in the ABI (languages with first class classes can choose to hide the this pointer and v-table).

Operator overloading (which forces name mangling or something like it) is tricky though. As is exceptions. I don't have answers to that.

> The point is the C lowest common denominator is a bit low. I'd like something that by dropping C we can be a little higher.

I don't think it really raise the bar much. Rust for example isn't an OO language so classes and v-tables won't be included. Go doesn't have templates so off they go. Operator overloading, multi inheritance, exceptions, garbage collection all go.

What your left with is essentially the C ABI. Being this lowest common denominator is why it's the the common ABI just as much as the success of the language.

Something like https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/... ?

Never ever ever gonna happen, although some progress can be made in areas: D is working (Quite a lot done already) it's way to being ABI compatible with C++.

What could be done, however, is to provide an ABI-as-a-library at each end then pass that with the C interface. Textual Specification, statically processed at each end into some kind of parser type thing. Still borderline suicidal effort, however, as not all languages (Go, I think?) treat the address space equally among a clusterfuck of other things. Any form of parametric polymorphism would be a disaster too, I assume.

This is a great idea (ignore the naysayers). It's been discussed before here and in the link 2 comments later: https://github.com/rust-lang/rfcs/issues/600#issuecomment-40...

Reminds me of COM.

Maybe, but COM's reputation is it is too complex. From what I can tell the reputation is well deserved.

Much of COM's horror comes from being an OO interop system designed to work in non-OO C. All sorts of manual reference counting. If you use COM objects in C# it's much less horrifying, although there's a certain amount of "body horror" involved in firing up one whole windows application inside another one. You can run IE in a cell of an Excel spreadsheet, but that doesn't make it a good idea.

Working with VB made COM really nice. Both for writing COM servers and consuming them. C# is actually a step back because it doesn't do the reference counting right and instead relies on GC to free objects.

> You can run IE in a cell of an Excel spreadsheet, but that doesn't make it a good idea.

Embedding a MS Works worksheet in a MS Works document however was sometimes a good idea in '95.

I've yet to see a better implementation than that and I find it really disappointing that the state of the art has moved backwards in this area (in addition to UX and performance relative to hardware specs).

I agree. Apple also had an effort called "OpenDoc" which they gave up on.

COM works pretty well in Delphi, .NET, C++ Builder, MFC, C++/CX and now C++/WinRT.

The problems and boilerplate usually affect those that insist in using the C level APIs.

It's too many things at once. The ABI is excellent and is used all the time, see DirectX, MediaFoundation, .NET, UWP, etc, they all use good parts of COM.

Scripting support (IDispatch, WSH) is OK despite a bit outdated.

Registration infrastructure (HKCR), GUI widgets (ActiveX), threading model (these apartments) and inheritance of implementation (aggregation) ain't good, way overengineered.

Compound document format (structured storage), RPC both local and networked, are just horrible.

Core COM is basically predefined vtable layout and object reference comparison rules combined with IUnknown for lifetime and type queries. It can be verbose, but it's very simple.

Stuff like OLE Automation is very much not simple, but it's not a requisite part of it all.

This one is my favorite:

https://msdn.microsoft.com/en-us/library/ms809983.aspx

I have think about this, how a REST API elide a lot of the problems in how integrate disparate stacks...

How have something alike? Maybe put a ring buffer as mediator and push/pull a binary encoding format (messagepack?). Probably need a little of C for the bridge.

Reply on news.ycombinator.com