Hacker News Reader

Comment by drewg123 | Hacker News Reader

Given a syscall that does nothing, a full round-trip under BSD would require about 40μs, whereas on a user-space Mach system it would take just under 500μs

This is still true, but to a lesser extent, on MacOSX (or at least was, a decade ago). I was writing HPC drivers for a cluster interconnect. So performance was critical. We had been using the BSD ioctl system to communicate with our drivers because we used ioctls in all our other drivers (Linux, Solaris, FreeBSD, Windows). I did some microbenchmarks and noticed that it was far slower than FreeBSD or Linux ioctls & complained. Apple suggested that I re-write the app/driver communication using IOKit, which is Mach based. The result was something that was twice as slow.

View on HN

Have you run a test recently?

Trivial mach IPC round-trip (same process) seems to be about ~8.5µs in my test, so around 4µs per call. I'm sure if you transferred port rights, memory, etc the cost would go up.

edit: I would advise anyone to run your own tests. Anecdotes aren't data and some things like syscalls are not nearly as expensive on modern hardware as they once were.

Sometimes things can get ridiculously faster than you remember. For example the original iPhone took nearly 200ns to do objc_msgSend() [1]. In 2016 modern hardware did the same thing in 2.6ns [2]. That's two orders of magnitude improvement. So saying "message passing is slow" is not a correct statement... it's almost as cheap as a C++ virtual method call.

[1] https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performa... [2] https://www.mikeash.com/pyblog/friday-qa-2016-04-15-performa...

What is a BSD syscall on the same hardware?

>Given a syscall that does nothing, a full round-trip under BSD would require about 40μs, whereas on a user-space Mach system it would take just under 500μs

Point of reference: I work in HFT, and it's possible to process a market data update from an exchange, reprice a complex financial instrument, decide whether to place an order, and send that order back towards the exchange all in under 5us. Another point of reference: raising one float to the power of another (a^b) could be done over a 100,000 times in the same time it takes for Mach to do a single syscall.

I really don't mean this as a knock on the work you do.. but what is the actual complexity of the operations you're describing? From what little I understand of it, all HFT operations are designed to be as simple as possible. Like.. I've read that the decision to the order engine could be as simple as a comparison between two numbers (some kind of arbitrage model), and the price adjustment could be just a simple addition/alpha adjustment. Basically, the offline/non-realtime stuff in HFT is way more complicated than the live/real-time stuff.

Still more complex than a no-op syscall, especially including message encoding/decoding.

That 500 microsecond number is from a 1993 paper on 1991 hardware (a 50 MHz 486DX-50). Your HFT code wouldn't load on it, and in 500usec it would be lucky to do a single float^float power operation.

> in 500usec it would be lucky to do a single float^float power operation.

The rest of your post seems OK, but this claim is almost certainly off. 2000 float^float operations takes a full second? Even on a 486DX, no way.

Ok I mixed up the DX and SX; at least the DX has the fp hardware. Still, many of the x87 instructions you'd want to use to implement pow() take hundreds of cycles on a 486, so I'd bet it would take tens of usec to do each pow().

The way XNU converged from its microkernel nature to a more monolithic one has led to all kinds of funkiness. You still have more syscall overhead (and VM but message passing isn't probably the culprit idk) but can't really trust Mach to separate kernel's internal systems like one would in a typical microkernel.

For presumably optimization reasons arbitrary pointers (rather than checked messages) are passed around different parts of the kernel. And that exposes some quite bad security issues from time to time.

So the obvious question is how it has changed over the years. What has apple done to pull latency down to a comparable time?

I doubt that they care. In the period where I worked on Mac drivers and kept close track of the Darwin sources (roughly 2003->2013) I saw very little in the way of performance improvements.

AFAIK, they still don't support 15 year old technologies like MSI-X that permit efficient multi-queue network drivers.

Have you ever built a project by hand on MacOS and then on Linux (or FreeBSD)? Have you noticed how absurdly, painfully slow it is running autoconf on MacOSX? That's because MacOS system calls are horrifically slow compared to Linux / BSD.

What does “horrifically slow” mean compared to a monolithic syscall? I haven’t run autoconf on linux in years.

Small number of minutes vs less than a minute. The last time I built something by hand in both places, it seemed to take 2x to 3x as long to run all the autoconf checks on MacOS. That's basically a fork / exec / open /close sort of benchmark.

(and general filesystem performance)

You should try it yourself.

We moved almost all development machines to Linux after someone demonstrated how fast Linux built some C project compared to osx and windows.

Spent a while thinking why I don’t see this much. I work mostly with higher level languages, so the beefiest things I run are probably webpack and the typescript language server. I wonder if maybe the problem is fork or some derivative effect. That, or I just don’t have any syscall heavy loads in my life, full stop.

It doesn't need autoconf to see this. I have seen it in a project with a simple make -j, compiling with clang in both places. Builds finish much faster on Linux.

To be fair this may be many things and not just mach syscalls.

It's `fork` in particular which is much slower. It has to fiddle with Mach port rights, doesn't do overcommit, etc. And autotools is very fork heavy.

Apple provides posix_spawn which is much much faster. Running /usr/bin/false 1000 times in the fish shell is nearly twice as fast (2.25s to 1.25s) when using posix_spawn instead of fork, on the Mac.

Autotools doesn't use posix_spawn because there's no benefit on Linux, but it is what Apple's frameworks use internally for process launching.

> Autotools doesn't use posix_spawn because there's no benefit on Linux

Wait, autotools don't make syscalls at all, do they? I thought it's just shell and make.

Is the problem that /bin/sh and /usr/bin/make use fork and it would help to have a Bourne shell and GNU-compatible make that used posix_spawn?

Reply on news.ycombinator.com