Hacker News Reader

Bias warning --- I'm part of the group doing this work. For an end-user, your arguments are fine. A single developer or team can indeed use whatever tool they want and adapt their code without too much difficulty.

The real challenge comes, however, as you try to build an ecosystem of libraries on top of multiple different APIs. You end up reinventing the wheel or struggling to maintain multiple versions of the same library.

Some cooperation is immensely helpful at the lowest level, and we hope ultimately to be helpful to the different tool builders in providing information and guidance about what is already a standard and agreed upon in the space.

This is not about establishing or forcing behavior. It is about documenting and clarifying best practices that have already emerged as well as potentially giving libraries a way to signal to downstream developers of additional libraries that they adhere to the standard.

Do you have any ball bark value you could estimate for the size of a bare bone common array implementation?

I'm dreaming of having basic numpy features in the Python stlib, but numpy is so huge, and I have no idea how much of it is the array itself, and how much are all the tooling around it.

I don’t really disagree with most of your points, but what you describe does not sound like it should be a committee with governance rules and member libraries tied to an RFC process.

What you describe sounds like you could just create your own single library like a Swiss army knife that contains adapters to rationalize the semantics if you want, even at the ABI level if truly needed.

Member libraries can do a best effort support for not breaking that adapter library’s way of wrapping them, and/or contribute patches that keeps the adapter in sync with new releases of the library.

This makes the adapter layer become totally opt-in and backported rather than required via consortium membership & RFC governance, and even leaves room for competitor adapter tools to coexist that might do things differently.

Ultimately my concern with a governance model is that it creates political power to compel the member libraries to do things that might go against the needs of their users for the sake of the consortium.

I just don’t see a reason why that’s more valuable than letting all the libraries do their own thing and let users choose or make their own custom adapters.

This example of array APIs is nothing like say overall HTTP protocol or IEEE floating point protocol, which are cases where a governing standard makes sense. This is nothing like that in my opinion.

So what would you have libraries like scikits-learn do? Support half a dozen array APIs until a 7th array implementation comes around? Support one, and force users to litter adapters everywhere and (often poorly) reinvent the wheel?

I'd very much welcome a unified array API initiative. It's been accomplished in Julia and it's incredibly empowering.

The fact that today you have to special case functions because the various array types are only loosely compatible is a nightmare, not a benefit.

Personally, I like how the Julia ecosystem coalesced around a Tables.jl package that gives some consistency to tabular data packages that choose to implement it. Row-oriented or column-oriented, doesn't matter! Using Tables.jl they can be interchangeable.

There’s a huge difference between “I personally like it when some libraries choose to adhere to a convention” vs “let’s bake this into every tool with a wide, bureaucratic shared RFC process.”

But I don't think people are forced into following right? There's still choice. But if I was developing something in Python, I would rather join than not join.

I’m saying as a consumer of these libraries, I don’t want joiners, I want lots of different APIs that make different trade-offs for each use case, and I will write (or use) adapters if I need to (that’s my responsibility as a consumer).

Just because I personally either do or do not get some value out of a shared API standard doesn’t mean you do or do not, so personal feelings of what one likes need to be set aside, so the service boundary is at the level of consumers choosing their own adapters and making their own trade-offs.

If lots of libraries sign up, we all lose. Much slower compliance-constrained development cycles, less freedom for libraries to break the shared API contract in ways many users would be super happy to abide for the sake of a faster / better feature. Instead of making these trade-offs library by library, case by case, it’s made for you with standardization compliance as the highest constraint, regardless of the user base for which that does / doesn’t work well.

This strikes me as a bit pessimistic. I think you are mostly against _bad_ standards than standards per se.

If the consortium does its job well it will produce a good standard that all the major libraries like: from a UX as well as a performance viewpoint.

If it produces a bad standard, all that will happen is that nobody will sign up. No libraries are going to be _forced_ to do anything.

That’s fair except I’d say even good standards can cause a big slowdown in feature releases, and not everyone values the adherence to the standard as much as the features. It still seems better to just spin out the standard into a set of optional adapters and let library maintainers and users pick their own trade-offs in terms of when to adopt breaking changes, how / whether to smooth over idiosyncrasies across multiple libraries.

Even a good standard has costs and this particular case does not seem like it has good arguments in favor of a wide standard, but many arguments against it.

No, it's easy for library maintainers to offer a compat API in addition to however else they feel they need to differentiate and optimize the interfaces for array operations. People can contribute such APIs directly to libraries once instead of creating many conditionals in every library-utilizing project or requiring yet another dependency on an adapter / facade package that's not kept in sync with the libraries it abstracts.

If a library chooses to implement a spec compatability API, they do that once (optimally, as compared with somebody's hackish adapter facade which has very little comprehension of each library's internals) and everyone else's code doesn't need to have conditionals.

Each of L libraries implements a compat API: O(L)

Each of U library utilizers implements conditionals for every N places arrays are utilized: O(U x N_)

Each of U library utilizers uses the common denominator compat API: O(U)

L < U < (L + U) < (U x N_)

Where the APIs are compatible anyway, but only the names or order of arguments are different, it sure would be nice to have a standard so you didn't have to look up whether it was count or len or length or size.

Disagree. Those are the super easy cases for me to create my own adapter that does that for me in a way that works for my use case, leaving others free to do it / ignore it at their own discretion, depending on their use cases.

Maybe one person creates various functional wrappers around the array structures to make composition of length or count work in their functional programming design. Someone else created mixins and their own derived classes that solve it with OOP designs, etc etc. Someone else only uses Jax and doesn’t care. Someone else uses infinite streams on top of these and so they never need length or count.

Forcing the same call semantics on all these users doesn’t make sense. Let them build their own abstractions if they need to standardize semantics for their use cases.

Perhaps a more direct Julia analogy would be the built-in AbstractArray interface.

There is simply no question of how (to use their example) `mean(::CuArray; dims)` should work, it not only follows Base's notation, but extends the same `Statistics.mean` function, so that exotic arrays can be passed to packages written before the package defining them existed.

As someone who's got some experience with this (but nowhere near as much as you have - I am an amateur), I'd like to strongly disagree with you here.

I find that slightly different APIs between libraries are a real energy-sapper, and having to do the mental gymnastics around them makes little sense to me. Yes, as a super-experienced programmer, you can remember each subtle difference in the syntax of each API, but I think many people find this difficult (me included), and it also leads to problems with code that's written when you've missed something or not used the appropriate part of the API.

One of the strengths of using SciKitLearn for me has been the interoperability across different ML models - by having the same API it means you can try something quickly without having to rewrite code, and can concentrate on the end results, not whether or not you're really telling it what you think you are because you've swapped an element of your pipeline out.

Learning the standard library of Python is a big task; there have been countless times when I've created something which is actually present, but I didn't know it was there, or how to look for it, and then 3 months later seen that utilised in some other code and had a "Why didn't I know that?" moment. I think having non-standard APIs across libraries will lead to this sort of situation, and a lot of reading documentation which would be obviated by the approach outlined in the article.

>Forcing the same call semantics on all these users doesn’t make sense.

I don't understand why not (providing, of course, they are all capable of having whatever is relevant conveyed to them appropriately?

What are the trade-offs that following this standard would involve (other than the delay due to bureaucracy that you mention)?

I don't see what is gained by the use of different method/function calls for what are the same method/function, as outlined in the table [1]

[1] - https://data-apis.org/images/array_API_comparison_output.png

I think a scary part of your comment is that since you don’t feel it would impact others use cases and you don’t see why some library might want different semantics, you leap to the conclusion that the burden of proof to prove these things rests with other people.