The stable ABI for all the major OSes at this point is expressed in C, or put another way, C is the language of FFIs at the moment. This is not something I am happy with, and I wish we could start moving away from it, but the result is that the way you describe the ABI of your OS right now is to provide a C header file containing function prototypes and structure definitions.
In principle, it's also worth pointing out that libc is really three separate (conceptual) things. There's libsyscall, which is the bare wrapper around system calls; there's libuss, OS services provided entirely in userspace and not the kernel (dynamic linking and some aspects of threading come to mind here); and finally, libcrt, which is the actual C runtime that implements the C standard library (e.g., fopen). It's really unfortunate that on most OSes, all of these are combined into one library called libc, although Windows actually separates these into different libraries (ntdll, which is not stable; kernel32-ish, which is stable; and msvcrt, which you have historically had to ship yourself!).