Hacker News Reader

Comment by inglor | Hacker News Reader

Hey, Node core person here (and the person who triggered the land) - we're super excited for this and would love help and feedback.

This is still experimental and we'd love to hear from the community what you'd like to see.

View on HN

Is there any chance to break spec and allow manual redirect handling? the fetch API makes a lot of sense in a browser, but imo this is a pretty crucial feature that undici's implementation lacks. [0]

Deno decided to break spec [1][2] so the following code works fine:

    fetch('https://httpbin.org/status/302', {redirect: 'manual'})
      .then(res => console.log(res.status, res.headers))

In undici this will succeed but with res.status 0 and no headers, as per spec. You aren't allowed to see the content of a redirect, like in a browser.

[0] https://github.com/nodejs/undici/issues/1072 [1] https://github.com/denoland/deno/pull/8353 [2] https://github.com/denoland/deno/issues/4389

A better idea would be to replace / improve the fetch spec. Handle redirects, set JSON as the default request content type, encode URI components for users, decode response bodies with JSON headers as JS objects, etc.

Right now most developers either write their own library to do these things on top of fetch or another HTTP client or use a third party library from npm.

A new standard would allow us to make http requests out of the box comparable to high level HTTP clients like superagent, axios etc. that have been in use for the last decade, since before fetch was conceived, with whatever benefits fetch provides (I think it's cancellable now?).

A new standard would need buy-in from browsers to actually be a standard. Browsers likely don't have a lot of incentive to make spec changes only Node is interested in (like making the whole spec more complicated from their point of view because of Node's (or Deno's) different security model).

The level of collaboration and good-faith we've been getting from spec bodies like WHATWG is very high as it is and we really don't want to push or abuse it.

Does it have to be a buy-in, though? You could agree on server-specific extensions to fetch and codify them in the same spec (so it doesn't get lost otherwise). Browser vendors wouldn't need to implement it but they will still keep an eye on it when working on future browser apis.

Codifying the redirect behaviour for server/"trusted" envs makes some sense with Deno/node doing the same thing. I'm not versed enough in the standards orginazation politics to say if that's viable though.

The rest of your "improvements" aren't that. They're opinions, opinions laser focused on a JSON centric api.

I'm sorry to say this, but not all of the web is one big JSON blob.

Some of us have to talk to SOAP (xml) services.

Sometimes it's weird rpc stuff with protobuf.

Or even just pulling down raw binary data and feeding that through an API/library that wasn't built for web streams.

Honestly, the amount of code you need to add over the top of a primitive like the Fetch API to get what you wanted (bar the redirect change) is trivial and minimal.

Can you share your setup for talking to SOAP services?

It would be useful for trusted browser extensions too, as well as probably Electon apps as well.

I don’t know if chrome apps are still in existence, but if they are the “server side” fetch spec could apply to them too.

> Browsers likely don't have a lot of incentive to make spec changes only Node is interested in

All the existing HTTP clients discussed in the previous post also run in the browser.

Why is this being down voted? This comment seems to be making a reasonable case.

In case you disagree, you may want to reply why instead of down voting.

Because fetch just landed in Node.js, making it a standard present in every context after like 5 years and he want to change the standard now.

Edit: also, you are being downvoted because the etiquette on HN are that you don't ask "why was X downvoted".

As mentioned existing HTTP clients had this behaviour back in 2012, the designers of fetch chose to ignore these cowpaths in favour of a more minimal implementation requiring the use of third party HTTP clients in both browsers and node, simply to have reasonable defaults and not repeat oneself, for the foreseeable future.

the value the js community gets from `fetch` being a unified standard is far, far greater than the benefit you as a developer would get from not having to add 15 extra LOC around `fetch`, that you'll probably hide behind a `fetchJSON` function anyway.

Point is everyone had a slightly different 150 LOC or a library to achieve essentially the same thing.

We could have stuck with ‘if index is not equal to minus one’ but we have array includes now for the same reason.

I didn't downvote, but my guess is its primarily a kneejerk reaction to the following line:

> set JSON as the default request content type

or something similarly subjective.

Downvoting is going a bit too far, but bundling a few individual opinions on defaults alongside broadly accepted (implemented in both Node & Deno after much discussion) features seems... presumptuous.

None of the things you mentioned are reasonable default behaviours, and are easy one liners. You don't need a library for any of that.

Thanks for your opinion, however as mentioned, or as you’ll be aware if you’ve used JavaScript, these are well worn cow paths that have been default in the most popular HTTP clients for a decade now.

Yes, this is a place where Node.js will diverge from the spec see discussion on the fetch PR https://github.com/nodejs/node/pull/41749#issuecomment-10254...

Is this not what redirect: “manual” is in the MDN fetch docs?

https://developer.mozilla.org/en-US/docs/Web/API/fetch

In that in the browser you are able to see that you have gotten a 301/302 response but can't see where the redirect would have sent you.

Oh right, because they don’t want cross-site scripts to be able to see redirected URLs since they could contain secrets. I wish we could completely do away with cross-site scripts and just have nice things!

Yea, so instead it would just encourage more 3rd party libraries doing random things on your site. This is what happens in native. Instead of embedding an ad in an iframe and isolating its damage you embed your ad service's library in your code and it spies on way more activity than it ever could otherwise.

I would also be okay (ish) with explicitly isolated third-party code execution, like your example of an iframe to a different domain. I'm pretty sure that should already be the case with iframes, in fact (you obviously shouldn't be able to embed an iframe to facebook.com on your website and then use your website's JavaScript to inspect the DOM on that facebook.com iframe).

It's pretty funny that such an RPC framework as the browser exists that gives the end user a genuinely decent sandbox, yet all it receives is criticism for its flaws. People will then happily install a screen dimmer or "productivity" tool with superuser privileges from a completely untrusted source.

Thank you for making "fetch" happen

underrated comment

Hi there,

> ...we'd love to hear from the community what you'd like to see.

In our tests, fetch on Deno was wayy faster than undici fetch that now has been merged into nodejs.

I couldn't figure out why that was case, but forwarding requests over node's http2 client (instead of undici fetch) then had comparable (but not as fast) performance as Deno (presumably because our hand rolled impl lacked connection pooling).

This is very new and the implementation _just_ landed today so it makes sense Deno would be faster with an API they have been working on for years.

I am confident that Node's implementation will _eventually_ have comparable performance.

That said: please do open an issue in the undici repo at https://nodejs.org/node/undici so this gets tracked.

That URL gives me a 404 – should it be https://github.com/nodejs/undici ?

Thanks. Filed: https://github.com/nodejs/undici/issues/1203

Deno's is probably written in rust via the tokio rust crate. This version of fetch in NodeJS is written in JavaScript. It would probably get upgraded to C++ in due time once it's mature enough.

Really happy to see fetch() merged into core! Looking at the PR, what are those WASM blobs in Undici and how is it that Node.js accepts a random very large compiled blob instead of asking for the source file + compilation step? https://github.com/nodejs/node/commit/6ec225392675c92b102d3c...

Those wasm blobs are Node's own llhttp https://github.com/nodejs/llhttp in wasm to speed up HTTP parsing. The project itself added as a dependency is also by Node https://github.com/nodejs/undici .

The question is totally legitimate but please assume core doesn't make "load random binary" level kind of goofs :)

Sorry def didn't mean in a bad way! I just wanted to ask/note that in the PR, and with the very little context I have, it seems like an arbitrary code blob so I was surprised, I'm 100% it'll be well documented in e.g. undici itself, and prob in other places.

Except that's exactly what Corepack does, just silently and opaquely on the user's system. At least, the moment it switches from opt-in to opt-out.

Its unverified binaries are just as good as random binaries, and silently replacing any pre-existing yarn/pnpm/whathaveyou without checking if it was perhaps a custom build is not exactly predictable either.

Other than Corepack, though, I think you people are doing an amazing job!

Hey this one change is going to positively impact my day to day life. Just wanted to let you know. Thanks for sharing your hard work and time.

I'm happy to hear that but I had a very small part in it all!

The person who took it through the finish line (and deserves the most props here IMO) is Michael https://github.com/targos

The people who worked most on Undici are Matteo https://github.com/mcollina and Robert https://github.com/ronag

The person to work most on fetch in Undici is Ethan https://github.com/Ethan-Arrowood

Also worth calling our James whose work on web streams in Node as well as events/cancellation helped drive a bunch of this.

I can name maybe 50 people who worked on this effort overall at some capacity (I am one of them - spending maybe 40 years on APIs to help enable this).

Note the job isn't entirely done and help is very much appreciated!

What does undici mean?

eleven

https://undici.nodejs.org/ "Undici means eleven in Italian. 1.1 -> 11 -> Eleven -> Undici. It is also a Stranger Things reference."

"undici" means "eleven" in italian. But I don't know the rationale behind the name

http 1.1

Quick question, judging from the commit alone this seems like a rather small change. I assume there's more to it though. I just wonder how come features like this that kind of seems obvious to include in the ecosystem takes quite some time to land? I understand the reality is more complex perhaps, so I am genuinely curious.

I hope you realize no disrespect, this will greatly improve the daily work for me since I use node at work every day.

Adding 8k lines is hardly a small change.

The commit adds undici (another Node.js project at https://github.com/nodejs/undici ) as a dependency and exposes its `fetch` - the code changes you see are probably just adding a flag :)

> I just wonder how come features like this that kind of seems obvious to include in the ecosystem takes quite some time to land?

I answered that below (check it out) but note how expensive adding a bad API is vs. asking people for one more `npm install` :) There is more discussion in https://github.com/nodejs/node/issues/19393 and in https://docs.google.com/document/d/1tn_-0S_FG_sla81wFohi8Sc8... a discussion from 2018 we had on it

Ok cool and thanks for the clarification, I imagined it was more to the topic than I realized.

Last question I have is will this be included in the next version of the LTS version or how do experimental features usually progress for someone who are unfamiliar?

Can't wait until I can use fetch in my codebase without any additional dependencies!

To add, unidici is an http client. Node already has http / https, but unidici is much faster and probably has other improvements, too.

They could have built a fetch interface on top of http, but they probably saw an opportunity to make more drastic improvements.

I had a quick look, the implementation is interesting

Node uses undici JS library. Undici includes a C HTTP parser that is used via webassembly. The parser is generated from rules that are implemented in JS/TS using llparse framework.

A native C binary there in the middle is quite surprising.

my guess is that this adds stuff to the global object which has backwards compatibility concerns.

And with every public API: Once you add it, you can't really hope to ever change it and even bug-fixes could be breaking some user's code (because they relied on the bug), so you have to be very careful to ship your public API as bug-free as possible.

Yes I could imagine and I have experience of developing apis with "bugs" that you kind of need to keep since people come to depend on it working a certain way.

But I can't help to feel like Node is progressing kind of slow compared to other languages / tools. This is just a personal impression from a bystander who is not really into the actual progression though. I have a feeling that node used to push new features all the time but in last couple of years have stagnated somewhat. Stuff like imports etc is still not really here.

I understand that developing an api that has such huge userbase as node is a massive undertaking that probably has issues that I can't imagine but it doesn't really help me in my day-to-day experience with it.

I think some of it is because Node doesn’t have very many ways to let people know about potentially breaking changes. A compiler (like Elixir) has an opportunity to communicate potential breaking changes to the user before the program actually runs, and a static type system (like Rust) makes it easy to detect such broken code and reject it outright instead of silently changing behavior. It’s harder for Node: where would the diagnostic even go? The console? Some apps use Curses, so this will break them, and many of them send their own logs through something other than the console, so nobody’s actually watching it.

Node also isn’t actually a language, so there’s a lot of stuff that Node developers get in new version, but it’s actually being done in the V8 project.

Meh, the js/ts ecosystem breaks things all the time. At least 2 weeks ago you couldn’t run typescript with ts-node, unless you explicitly enabled highly experimental loader flags.

And this is no obscure corner case, ts-node is huge

Where can I find an explanation about what is Fetch and why it should be in Node.js?

This is fetch: https://github.com/whatwg/fetch https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

Here is discussion about it in Node with some (pretty old) discussion https://docs.google.com/document/d/1tn_-0S_FG_sla81wFohi8Sc8...

It looks like it is a new standard: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

I thought it was a Node popular library.

It’s “new” as of 6 years ago or so.

Fetch is a popular browser api to download and request stuff from the internet. Having the same api both in browser and in node allows easier re-use of the same application code or libraries on frontend and backend.

fetch is a "modern" (ES6 era, so not brand new) API for making HTTP requests, to replace XMLHttpRequest (the older method that IE pioneered).

It's promise based (thus easier to integrate into code than older callback styles), and designed to give some better options around CORS and handling responses.

> Why it should be in node?

There is a push to adopt some certain "web apis" (APIs that emerged in web browsers) to increase compatibility, reusability and reduce cognitive load when working with both backend and frontend javascript.

Having a common low level call like fetch() means that libraries that are built atop it (SDKs to talk to services, or apply common middleware like JSON:API, oAuth etc) can be shared between node/deno and browsers.

I do have a question that's only tangibly related to fetch, I was wondering what Nodes stance is on the seemingly increasing gap between the NodeJS and browser implementations of the JavaScript engine.

It's still very common to see Node modules that use require(), for example, but would otherwise work flawlessly in a browser where there is no require() support (without using shims and other libraries).

I don't speak for the project a whole :)

Node has supported ESModules (`import`) for a while and you can use it (either by naming your files `.mjs` or setting `type: module` in your package.json.

There is a solid interop story between require/import and you can mix if you want.

`require` is going to be supported "forever" so code doesn't break and both module systems work.

As for the general question: Node is committed to supporting modern JavaScript and to not allow such a gap to be created. If you see a place Node doesn't do a great job at that please tell us!

However note that ESM in Node comes with drawbacks that prevent end-users from relying them in various situations. Those will be mostly solved once loaders become stable, but until then it's still advised to ship packages as both CJS and ESM.

How do you do both?

It's very easy to re-export ESM (import) as CJS (require) and vice-versa. The main issue is that ESM by default

For example to use `require` inside ESModules you would do:

```mjs import { createRequire } from 'module'; const require = createRequire(import.meta.url); require('./whatever-in-cjs'); ```

There is a reason this isn't "by default" though since ESM doesn't "silently" interop with CJS to not make writing universal code harder.

Usually you setup a transpiler to target both. For example, my packages are bundled with rollup towards both cjs & esm:

https://github.com/arcanis/clipanion/blob/master/rollup.conf...

Helpful, thank you!

> I was wondering what Nodes stance is on the seemingly increasing gap between the NodeJS and browser implementations of the JavaScript engine. It's still very common to see Node modules that use require()...

"Seemingly increasing gap" - how did you come to that conclusion? If anything the gap has reduced since the availability of ES modules. I see more and more libraries preferring ES modules, but ultimately that's up to the library developers.

Huge thanks to the node team for adding this, I’ve been wanting fetch in node for years now! Installing node-fetch for every project was getting kind of old haha

Thanks a ton for this. Fetch API is elegant and it working on node natively will so many edge cases we need to handle currently. Thanks a ton, please know that this will positively change our life and tens of thousands of other developers worldwide.

Great work!

Curious to check but quick question: does it support upload progress? If not, is it considered?

You can upload a stream and monitor that for upload progress - so sure.

If you upload a string for example you would have to take an extra step to get upload progress (make it into a stream and monitor that).

There is additional (unrelated) talk in the fetch standard itself to provide this sort of functionality as part of `fetch` which would save the middle step.

yes, yes, yes.

This was a huge pain for us. We develop an SDK that needs to work on Node, browsers and React Native.

I am super happy to hear about this :)

Does it mean we have a built-in FormData in Node.js core for multi-part requests?

Is there support for timeouts? It's the main reason I use https://github.com/sindresorhus/got

You can use timeouts through `AbortController/AbortSignal` - eventually it'll even be built in with `AbortSignal.timeout` which is currently under-works in the spec level.

No, thanks. I'll stick with got.

Do you have any sense of the expected performance of web streams as compared to the existing node.js streams?

I wish if Node had a way to enforce any case in its requests.

This is awesome, thank you. Excited to take it for a spin.

pretty great. I've been using undici for a while now and very happy with the speed and experience.

FormData and Blob/File support?

Reply on news.ycombinator.com