Show HN: I've updated my Linux System Call website and API | Hacker News Reader

Show HN: I've updated my Linux System Call website and API

syscall.sh

238 pointsguitmz58 comments

I first released those in back in 2020 and now I finally added more architectures, Kernel selector, syscall descriptions and a few other niceties. It should work a bit better on mobile devices too but I am no CSS expert.

The API is at https://api.syscall.sh

Open articleView on HN

It’s a side issue, but I’ve long thought that the kernel should duplicate every existing syscall that takes a null–terminated string by pointer with an otherwise identical syscall that takes the string by pointer, capacity, and length instead. It would be a huge boon to programs written in modern languages. I keep meaning to search the kernel mailing lists to see if the idea has been proposed before, but I never quite find the time.

Modern languages .. or more specifically .. modern compilers for those languages .. can surely sustain a pointer, capacity and length wrap to pointer until null ..

That’s not the point. The point is to avoid the work of repeatedly making sure that the null terminator exists, and of finding the lengths of strings by scanning them looking for that null character.

Why also capacity?

What I think would make sense is a change to the C API where the error code is an out parameter and not that ugly thread local errno. But I guess that's not big enough of an issue for people/languages that use libc as system call interface.

You’re right; technically read–only access to a string only needs to know the pointer and length. I didn’t take the time to polish my message very much.

Errorno is stupid and annoying too, but I think that’s mostly a libc thing rather than a kernel thing. The syscalls themselves simply return error codes in a register as a return value. Worth fixing though.

Would also recommend using commands like:

echo '#include <sys/syscall.h>' | gcc -E -dM -xc - | grep -i "read"

To get the real values on your PC to make sure.

I had LLMs give wrong numbers or constant definitions several times but getting it to generate a command like this and seeing it for myself works every time.

Also chromium docs has a table:

https://www.chromium.org/chromium-os/developer-library/refer...

It would be very useful to just make this website directly link to the syscall number.

The website content comes directly from Kernel sources so should be accurate. The chromium page is what I used to use until I made this website in 2020 but it's very outdated I think

On x86_64 and x32, the syscall number goes in eax, not rax. The upper 32 bits of rax are ignored and can be anything.

Good catch, thanks! Since 5.14 (I think?) the kernel only reads eax (before that it used all of rax and returned ENOSYS if the upper half wasn't zero), so I'm adding a note about it to the x64 and x32 conventions

According to [1], the kernel has flip-flopped a few times on this, but the original behavior was to ignore the upper 32 bits.

[1] https://lore.kernel.org/all/20210518191303.4135296-5-hpa@zyt...

This is really great resource, didn't know about it. But one question not related to the website above - why the syscalls vary per architecture if the linux API is the same?

Or is it I am confusing it with the C standard library/POSIX and assuming that it should be the same everywhere?

The Linux kernel API is the same everywhere, but the syscall ABI (numbers, calling convention, entry paths) is specific to a architecture, while it's the C library that wraps those details into the portable POSIX API we code against

Also newer archs (riscv) don't support certain syscalls (open, fork) they are still functions you can call under your standard musl,glibc but they inernally call openat, clone.

I see, but why were they numbered differently? Why not use the same numbers?

As far as I know, the numbers were assigned independently by each port's developers in the 90s as they added calls in their own order. Then it became a frozen ABI nobody could ever renumber, and couldn't have been identical anyway since arches need different syscall variants for things like 32bit vs 64bit types

There are also a couple of syscalls which are truly architecture-specific - like iopl(), which is x86-only. (It controls access to the IN/OUT instructions, so it's not applicable anywhere else.)

I also made a CLI for consulting the syscall API when I originally released the website. I think I can tidy up the code and make it public too. It uses a embedded dataset so it does not have to query the API endpoint all the time

Somewhat related - I am working on a dual Pentium III retro build (first retro build) and completely forgot that this would be 32-bit and won't have niceties like SSE2. Having some fun going down the rabbit hole of IDE, SCSI interfaces and AGP graphics. Can anyone recommend some OS installations to play with? So far I am thinking Haiku, ReactOS, NetBSD and antiX or MX Linux. At some point I am also going to simulate a Novell Netware deployment.

Good list, maybe SerenityOS too?

NextStep, GEOS, Quarterdesk/X come to mind ..

Also various versions of OS/2, or for something completely different, iRMX (PC versions that co-install with DOS or Windows).

For NeXT OSes, OPENSTEP 4.x also ran on x86.

Just to say, this is very cool!

Thank you!! CLI coming soon too :)

That website messes with my graphics for some reason.

Always state your web browser name, version, OS, and re-run it without any plugins enabled.

This here is "Hacker" News, not "Newbie" News.

I am not familiar with the internals, so excuse my question, but how does one builds this? Separately, how often can it get stale?

It was stale for many years because I had a script to generate the tables for me that ran manually and then I had to convert the data into SQL so my API could use it (and also convert to html for the frontend). I changed the script to generate the dataset from Kernel sources as JSON instead and the API loads it and makes it available.

Then the frontend is refreshed from the API weekly via a cronjob with some simple Go templating

How often does it break or needs changes to this job and template? I am curious on how can this datasource be trusted and to what extent

It was not a problem of the template but rather that I never really "finished" it before so it required manual intervention every time I needed to update the list. I finally put some time into it and it is automated and easily extendable now, so it should continue to work with minimal/no effort.

The data comes directly from the official Kernel source code so it is as official as it gets, I don't change anything just format it in a way my website and API understands

This is very handy, thank you. Currently learning aarch64 assembly

hehehe can't wait to get syscallin'

Muito foda TMZ!

É nois!!! Valeu!

If syscalls are so similar, for instance, like 'read' and 'readv' or 'pread', 'sendmsg' and 'sendmsgv' or 'fork' and 'vfork'.

Does it still make sense to term or classify them as two distinct syscalls?

It definitely makes sense to have a section "similar syscalls on this architecture" on the details page.

There's a "Same syscall on other architectures" section already, maybe I could add the Similar section next to it. Edit: but this means it needs a way to actually know they are similar apart from only their names

This maybe unpopular idea if you've avoided AI thus far, but it does seem like an LLM would make short work of that task.

I wouldn't call any of them similar, except for 'pread' maybe. Each one has a lot of subtle semantics. Either way, it's syscall.tbl that determines whether they are distinct syscalls, not the author's page

I'm just noting this because it makes the number of Linux syscalls seem very exaggerated at a whole around 400 syscalls.

But, anyways, not like it matters that much but still.

400 distinct syscalls is a very large number of syscalls if you generally presume they're entirely distinct.

I think the number is mostly meaningless even if we ignore whether they're distinct or not. Consider ioctl() - it's a single syscall but if I remember correctly the number of subcommands is in the thousands.

That's right.

Moreover, for a large number of traditional Linux syscalls (including the many variants of read/write/send/recv) there are now better alternatives that are invoked as subcommands of a single syscall: io_uring_enter.

io-uring is great for this.

However, I am bit skeptical as to whether io-uring will ever support all Linux syscalls.

At this point I am convinced that io-uring primarily supports I/O API / syscalls. That would mean syscalls relating to disk and network I/O. While this covers a lot of syscalls it most definitely doesn't cover all.

Though, to give some credit her, it does also cover to syscalls relating to kernel features like futex but yeah, for the most part it doesn't cover much that doesn't have to do with I/O.

Which is obviously not a problem but raises the question of whether it ever will support other kernel features.

The io_uring interface is preferable to classic syscalls for any kind of syscall that requires the kernel to perform a significant amount of work or to wait for some external events, so a long time could pass until the completion of the work or until the occurrence of the external events, and possibly for some auxiliary syscalls that modify settings relevant to the previous syscalls.

Many of the most important syscalls belong to this class, but most of them already have io_uring alternatives. Many other syscalls are not of this kind, so for them there is no reason to replace the traditional syscall interface.

For the syscalls appropriate for io_uring the essential characteristic is that their completions can occur in a different order than the submissions of their requests, which is why all previous methods of handling them were inefficient, because they required the user program to waste time with superfluous activities, instead of just picking the completion messages from a queue.

Where this out-of-order execution is not needed, invoking the syscalls in the same way as you invoke any user function is fine.

The initial implementation of io_uring lacked the alternatives for many important syscalls, but since then most of the holes have been filled.

Besides more obvious syscalls of the read/write type, which are used for files accessed in the classic way, in order to reach maximum performance with memory-mapped files the old "madvise" syscall must be replaced with IORING_OP_MADVISE, used with Linux-specific advices (MADV_COLD, MADV_PAGEOUT, MADV_POPULATE_READ, MADV_POPULATE_WRITE).

Yes, the ioctl has been notorious for sneaking in arbitrary amount of functionality into the kernel space, and serving generally as a kitchen sink: "You can rarely convince Linus to make a new syscall, but making a new ioctl is much easier". That's still one syscall though.

And how do you even compare how the distinct the syscalls are (e.g., pipe() and open() are about 70% similar)? And after that, what do you do with this metrics? How do you sum it up? Again, pipe() and open() are 70% similar — does that mean we have 1.3 syscalls instead of 2?

I would count "pipe" and "open" together as 0 (useful) syscalls, because both have been obsolete for a very long time (even if they occupy 2 slots in the syscall table and those 2 slots will remain occupied forever).

These 2 syscalls have a history typical for Linux syscalls, where frequently new syscall variants make completely obsolete their predecessors, but the old syscalls must still remain supported forever by the newer Linux kernels, to allow running legacy programs.

"pipe" has been obsoleted by "pipe2" (on 2008-10-09), which in turn has been obsoleted by "IORING_OP_PIPE".

"open" has been obsoleted by "openat", which in turn has been obsoleted by "IORING_OP_OPENAT".

Most users do not invoke Linux syscalls directly, but they use the standard C library, which has frequently shielded the end user from the evolution of the Linux syscalls. For instance, you might use the functions "open()" and "pipe()" in your C program, but the C library might invoke instead of those the more modern syscalls "openat" and "pipe2", after adjusting their parameters.

I think you're missing the point I'm trying to make. Both pipe2 and openat has about 70% similar functionality (and so do the IORING_OP_PIPE and IORING_OP_OPENAT operations) — really, the only difference is that pipe2 returns two new fds instead of just one, and that the file opened is not a regular file. So, are those 2 syscalls or 1.3? What about triples or quadruples of syscalls with overlapping functionality, do we go full inclusion-exclusion formula on them? What about that 70% similarity estimation — how do establish that? Do you count the similarity/overlap of the underlying implementations? Only of the user-facing functionality? Both, somehow? Something else entirely?

Or, you know, we can instead not engage in this subjective quagmire and look at the actual, factual number that undeniably exists. It may not be very useful for any practical purpose except for pointless dick-measuring between OSes with radically different APIs and internal architectures — but neither is this trying to measure syscall similarity, really.

I think it's a bit unfair to mainly focus on the API surface while ignoring a lot of internal details. Linus has talked about how Linux will often have massive amounts of internal code change between releases, while the API surface stays the same. So yeah, the API might be old and crufty, but it's also been around for 30 years. And as long as the API is sufficient, it's the internals that are a lot more interesting. Radically redesigning the API every few years would just accelerate towards XKCD 927, because of backwards compatibility.

The number of Linux syscalls depends on the architecture and on x86-64 there are over 450 Linux syscalls.

Nonetheless, about one third of those syscalls are obsolete (being replaced by newer improved syscalls), even if a large number of programs are still using them, and about one other third of them are rarely used in the environments familiar to most Linux users, like personal computers and small servers.

The obsolete syscalls inflate a lot the number of Linux syscalls, because there are a lot of syscalls that were originally defined with an inadequate set of parameters, then they were fixed by adding parameters or changing the meaning of the existing parameters. There are many syscalls were even 3 or 4 attempts were needed until reaching an acceptable definition for their parameters.

New programs should use only the newest improved syscalls, but the documentation of the Linux syscalls in various places is usually not up-to-date itself, so a lot of people are still using older syscalls, without being aware that they are suboptimal. So the obsolete syscalls will have to be supported forever by the Linux kernels (though in most cases when an obsolete syscall is invoked it just fills the parameters with adjusted values and then it jumps to the current syscall).

I'd love to learn more about the syscalls that took 3 or 4 attempts.

Well, from a quick look, Linux has dup/dup2/dup3 and clone/clone2/clone3 . I don't see anything that took 4, but there are things like accept/acccept4 and where the second one takes four arguments and wait4 which takes 4 arguments, but wait3 was already obsolete before Linux or FreeBSD started numbering syscalls. (I'm lazy and only looking on systems I have)

It's not some Linux specific issue either. FreeBSD has three versions of statfs/fstatsfs/truncate/ftruncate.

A lot of the file related syscalls also have a XXXXat (ex: open/openat) variant taking a directory reference instead of using cwd.

Well, there really is about 400 syscall numbers allocated, and each has its own entrypoint in the kernel. That's an entirely objective criterion as opposed to arguing "well, sbrk and mmap (or recv and read) are kinda similar, maybe we should count them as just one syscall instead of two", because in the end we can might as well say "well, technically there is only one syscall, it just has an additional seventh argument that serves as the 'function denominator', like the ioctl's request parameter, you know".

I'm obviously not going to discuss this argument further.

Trying to counter a logical argument with a technical argument is pretty lame and doesn't make for a healthy conversation.

Uh, sorry, what? Trying to measure similarity between syscalls is a) impossible, b) pointless, so obviously I'd really like to understand why someone would want to do something like this — but I'd rather not speculate about what and why you think and believe, and instead to just straight hear from you.

If you're interested in why I think what I think — there's a sibling comment where I tried to articulate my position. I'd like to hear about yours, if you don't mind.

P.S. You didn't actually offer any argument, logical (?) or technical (?!), in your original comment, and I think you're actually have the meaning of those words reversed: "logical argument" appeals to logical relations between the abstract concepts, while "technical argument" appeals to the underlying technology (the kernel implementation in our case) and tries to deal with particularities of reality. So if anything, my argument was the logical (semantical?) one, not that it really matters.

Yeah, sure, but the obvious fact that is a lot of syscalls are 'family' relatives.

And they wouldn't ever count as distinct syscalls.

> they wouldn't ever count as distinct syscalls.

According to whom, exactly?

If a syscall morphs into several slightly variant syscalls, then there's no way these are distinct syscalls.

If a syscall is designed to counter the flaws have been observed in another syscall, then there's no way this is a distinct syscall.

If a syscall is designed to add feature to an another syscall, then there's no way this is distinct syscall.

And so on and so forth.

If you actually look at Linux kernel code, you will quickly note that a lot of these related syscall even happen to share a large amount of implementation code.

So, yeah

  > 400 distinct syscalls is a very large number of syscalls

Compared to what? It seems windows has 2000+ syscalls all combined (nt+win32). For just ntos there are 490.

https://github.com/hfiref0x/SyscallTables

https://github.com/j00ru/windows-syscalls

And does the number matter? Why can't it be 1000, 2000, 4000. In my opinion the interface is free to be defined as long as it is documented properly.

Yes because they have a different "syscall number".

Doesn't really matter if it has the same name or similar (or identical) function. This website seeks to describe the syscall ABI and that means that each number should get it's own entry.

And there is no identical syscalls in the list, they all have slightly different behavior.

It would be nice to make a banner or warning for deprecated or surpassed syscalls. For example (as mentioned in other comments) you should probably use pipe2 instead of pipe and similar other examples.

Reply on news.ycombinator.com