Show HN: Qew – a tiny queueing library written in TypeScript | Hacker News Reader

Show HN: Qew – a tiny queueing library written in TypeScript

github.com

14 pointsarrow700014 comments

I’m sure there is a good reason but I never got why nodejs/js doesn’t have channels something like Go https://gobyexample.com/channels

Countless times I have wanted endpoints to send data at certain intervals via ws/http/etc and the have the node app upload them somewhere else in an orderly queue but that doesn’t really seem possible. But with this library or similar you can.

It just seems like a common use case that isn’t implemented natively.

The observable pattern is on track to become a standard. See rxjs

According to the TC39 ECMA262’s repo [1] (ECMA262 is a committee responsible for planning and creating new syntax that goes into JavaScript) Observables haven’t been brought up at a meeting in over three years and they are still at stage one of the four stage process. I’m not saying there’s not hope I’m just wondering if it will end up happening. Either way I’m fine with or without Observables as async iterators, Node streams, and WHATWG streams are already pretty powerful. In fact the creator of RxJS made a library based off iterators called IxJS that looks interesting and hopefully eventually iterators in JS will have iterator helpers [2] where you can chain things off of them (currently stage two and last talked about last year). Also another thing to keep your eye on in the standards process could possibly be emitter [3] it’s also stage 1 (like Observable) but was talked about last year as a kind of push based replacement for Subjects in RxJS/Observable land.

[1]: https://github.com/tc39/proposals/blob/master/stage-1-propos...

[2]: https://github.com/tc39/proposal-iterator-helpers

[3]: https://github.com/tc39/notes/blob/master/meetings/2019-06/j...

Nice work. I've used [1] bottleneck in the past to do this, but it's very complex and bloated for simply limiting concurrency. Might try yours out soon!

[1] https://github.com/SGrondin/bottleneck

Please do! I'd love some feedback or suggestions when you do :)

This is my first published NPM package. It has a very simple API but I'd welcome any feedback and suggestions. Thanks for looking!

I’d recommend tests, also you may be able to return the promise directly instead of wrapping it in essentially a promise constructor. It could also be more efficient with an (ordered) set or linked list instead of an array. Otherwise looks fine but I’ll keep using rxjs ;p you should check it out. If you search my name on YouTube I made a whole playlist. The official docs are good too.

Thanks for the feedback!

I've spent a lot of time trying to add tests but ran into a ton of problems trying to get Jest's useFakeTimers and promises to play well together. I will take another crack at to see if I can work around the problems this time.

Not sure I understand what you mean by returning the promise directly?

Could you explain a bit more why using sets or linked lists would help with efficiency here?

Thanks I'll check out your playlist!

Re: jest, make sure to flush the event queue! Eg await Promise.resolve() which causes the test to pause while all promises queued by the code under test up until that point flush

Why use this as opposed to something more robust and common like rxjs?

Not sure you can limit concurrent async functions easily in RXJS. This seems to be a much nicer API for that use case.

Easy != forthcoming. You can do it with 2 operators, defer and mergeMap. There’s a learning curve to be sure, though... since rxjs uses less commonly understood academic/traditional concepts.

Full disclosure I haven't used rxjs myself but from this [1] SO answer it seems pretty complex to limit concurrency of functions in a queue.

Qew on the other hand has a much simpler API: you instantiate a queue with the max concurrency and delay set right in the constructor and then all you need to do is push async functions onto it and forget about them.

The queue will take care of limiting concurrency and executing the next function when the first one has completed.

You can use the same queue for your entire application or instantiate different queues for different REST API endpoints for example; setting the max concurrency and required delay between requests for each API depending on their individual rate limits.

[1] https://stackoverflow.com/a/26126750/3229534

Fair enough. It’s dead simple in rxjs too, but they lack concrete examples which has held back the library. Thanks for the detailed thoughts.

FYI that SO answer is outdated and does a superset of what’s involved in OPs lib. All you need is defer and mergeMap really at least with rxJs v6+

Reply on news.ycombinator.com