The underlying NT (microkernel) api (as found in used-to-be-undocumented NTDLL) returns STATUS_XXX from functions. With STATUS_SUCCESS being 0. This is still consistent as far as I know.
Win32 API, technically being a "subsystem" on top of NT, exposed to the user whatever convention its users were used to. Coming from Windows 3.1, it used BOOL (int actually) where FALSE was 0 and TRUE was 1, in a (mostly) consistent manner for success/failure.
An additional wrinkle may be APIs that return a HANDLE (or address for memory allocators) or NULL, which again if you treat as ints return non-zero for success and 0 for failure.
Posix Subsystem, on the other hand, could return 0s and non-zeros as desired.
So, if you knew Windows and thought about the API you were calling it made sense if you thought in terms of the types defined (BOOL, HANDLE, etc. for Win32) instead of 0 and non-zero.
No idea if it this is still consistent over the surface of the entire exposed API set now. So much accretion of cruft over the years under the eyes of an uninterested leadership has led to a lot of fragmentation and hence deterioration in the developer experience.
Except some Win32 API functions return INVALID_HANDLE_VALUE (= -1) on failure! Of course, there is a Raymond Chen blog post: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40.... Win32 error handling is indeed a mess...
Unix APIs are a couple of syscalls and library functions. If you want to achieve more (like video, audio...) you have to resort to "optional" libraries that may have bad APIs too, but they aren't core Unix and can be replaced by something that is hopefully better.
A valid criticism might be that one shouldn’t use true to mean success.
In cases where there is only one successful result, 0 makes sense. Then nonzero can be a range of error codes. In cases where there is multiple successful results and only one failure, 0 means failure; malloc() is the stereotypical example of this.
This is unfortunately true of Unixes as well.
The one "Unix" API I know that returns > 0 on error is pthread, which returns +errno directly (but still 0 on success).
Which APIs return 0 on error? I can't think of any.
My guess is this is usually true in practice, but in C and C++ dereferencing a null pointer is UB so you really can't assume that.
What I'm saying is there isn't an ergonomic issue with NULL error return APIs. You might prefer algebraic error types (in other languages). I'm not even sure I do. NULL return APIs are clean and simple. If you do (accidentally) forget to check for NULL return code, you will for sure get a segfault on first access in practice (not on embedded platforms maybe). The compiler can't remove any following code unless it can prove only NULL will ever be returned. Don't fall trap to UB FUD.
It can reorder code, though. For example, it can change
int *x = malloc(sizeof(int));
*x = 42;
if (x)
launch_the_nukes();
to int *x = malloc(sizeof(int));
launch_the_nukes();
*x = 42;Most of the UB FUD are contrived examples that hardly happen in practice. What I'm saying is not that you should write bugs, but that bugs (especially segfaults) will immediately turn up in practice.
Looks like the RtlRunOnceExecuteOnce API and structures appeared in Windows Vista.
So one would have to do some code archaeology and dig into the Windows Git repo to look at the commits for RtlRunOnceExecuteOnce and see what the typical use cases were. For some reason, returning 0 for failure made sense. Given the pervasiveness of NTSTATUS, I have no idea what made them choose a non-NTSTATUS return value. Maybe there was another callback-type API which did something similar?
If one dev thought that RtlRunOnceExecuteOnce required NTSTATUS, then I'm sure other devs will have thought the same thing so I'd expect more such bugs in the current codebase and in the future. That's why Microsoft static code analyzers (PREfix, PREfast) would flag such footguns.