undici: Can't remove default headers. Can't overwrite `"sec-fetch-mode"`, `"connection"` headers.

Enforcing CORS/browser specific rules in a serverside request context is non-sensical. These rules exist in browsers for security reasons to protect the user.

Fine to default to them if the spec requires that, but there definitely needs to be a mechanism to override the default headers. Core constructs should be configurable to enable a wide variety of use cases.

I can simply use http or curl to set whatever headers I want. But why would we handicap the standard lib and require dropping into lower level constructs for this? There is 0 security benefit to having this in fetch for serverside contexts.

Having a node specific mechanism to override default headers does not violate the fetch spec/api or prevent isomorphic fetch code.

So, I assume the next things will be implementing of CORS, CORB limitations, restricting of the access to “cookie” header since it’s a browser API? [sarcasm]

You know, there are people out there asking Node to do all these things (like CORS support) and some server runtimes like Deno with the concept of an origin are in fact implementing some of this stuff.

So this is not as out of the question as you may assume.

Also, please avoid sarcasm/snark in the tracker. We don’t know you well (yet) and it’s super easy to take things the wrong way in these situations without a lot of prior context (which we don’t have yet!) and it’s important to me that your contribution (feedback) on what node does here doesn’t get overlooked because of that.

That said I tend to agree node’s fetch should allow overriding these things unlike the browser’s. I also absolutely agree with Robert it’s not clear cut. I can ask in a forum of server runtimes I participate in but if you want to move this ahead it’d be great to do what server runtimes that implement fetch do (Deno/cloudflare/shopify etc) as well as what user-land implementations (like node-fetch) do.

undici is not a browser. Don’t overengineer it.

Easier said than done. Knowing what parts of the spec to ignore or not is not that easy. To keep things simple with the limited development time we have (I’m doing this work for free) it’s much easier to just follow the spec verbatim, i.e. the aim here is to match the browser as closely as possible and not bike-shed over what is right or wrong; the spec/Chrome is right (with few well discussed exceptions).

undici is not a browser. Don’t overengineer it.

fetch is a browser API. If you don’t want/need the browser behavior then don’t use the fetch API. We have much better API’s for node, e.g. undici.request.

I agree that this should be reopened.

I just spent an entire day narrowing in on why my proxy requests were returning 400, but working with node-fetch. Turns out removing the sec-fetch-mode (or changing it to navigate) fixed my issue.

The goal for fetch() is to be as close to the WHATWG spec as it’s possible right now. If you need lower-level APIs use undici.request().

Then what’s the point? The only reason for using this lib is because its now native.

You can get around this whole thing with something like the following:

import { ProxyAgent } from 'undici'

class CustomProxyAgent extends ProxyAgent {
  constructor(opts) {
    super(opts)
  }

  dispatch(opts, handler) {
    delete opts.headers['sec-fetch-mode']    
    return super.dispatch(opts, handler)
  }
}

export default CustomProxyAgent

Such a weird goal to be 100% spec compliant for a browser fetch.

Could this be reopened? It was incorrectly closed by #1829, which only addresses connection. sec-fetch-mode still can’t be unset or changed, as far as I can tell.

Yeah, I also would like if Node.js have some permission list in package.json like it is in Web Extensions, Android Applications: My comment in the topic about the recent RIAEvangelist/node-ipc case.

While sec-fetch-* contains “fetch” word, it’s not a part of Fetch API. sec-fetch-* headers were added in browsers much later than Fetch API. The modern browser send sec-fetch-* headers with any request, not only with XHR made with fetch function.

Also, why it sends sec-fetch-mode: cors?

cors

What if I do a request to the server from the same process where I host that HTTP server? Is it still “cors”? What even CORS means in a Node.js application? What about other sec-fetch-* headers? The browsers currently send 3-4 “sec-fetch” headers with every request.

I see default adding sec-fetch-mode: cors header is meaningless, and even annoying if I want to simulate a fetch request from a browser before 2019.08 (Chrome 76).

If any one want to simulate a request from the modern browser (with absolutely the same headers, and the same header’s order), he can (and should) add all required headers by yourself.

About manual removing the other default headers:

how that would look

Somehow. It can be a property with the array of string (header names) which should not be added by default, for example.

UPD (2022.04.08):

For example, to export a setDefaultHeaders function. It would pretty convenient. Note, it would have the global effect. (As well as using setGlobalDispatcher currently)

I think for people Fetch API is about “promises” and “streaming”, it’s about Request, Response, ReadableStream and Headers objects. But it’s not about which headers a browser adds.

Yeah, I also would like if Node.js have some permission list in package.json like it is in Web Extensions, Android Applications: My comment in the topic about the recent RIAEvangelist/node-ipc case.

While sec-fetch-* contains “fetch” word, it’s not a part of Fetch API. sec-fetch-* headers were added in browsers much later than Fetch API. The modern browser send sec-fetch-* headers with any request, not only with XHR made with fetch function.

It’s in the fetch spec.

Also, why it sends sec-fetch-mode: cors?

Because the spec says so.

cors

What if I do a request to the server from the same process where I host that HTTP server? Is it still “cors”? What even CORS means in a Node.js application? What about other sec-fetch-* headers? The browsers currently send 3-4 “sec-fetch” headers with every request.

We haven’t fully implemented the spec yet so the other headers are missing.

I see default adding sec-fetch-mode: cors header is meaningless, and even annoying if I want to simulate a fetch request from a browser before 2019.08 (Chrome 76).

We only target latest version of spec. Old browsers are out of scope.

If any one want to simulate a request from the modern browser (with absolutely the same headers, and the same header’s order), he can (and should) add all required headers by yourself.

About manual removing the other default headers:

how that would look

Somehow. It can be a property with the array of string (header names) which should not be added by default, for example.

PR welcome

I think for people Fetch API is about “promises” and “streaming”, it’s about Request, Response, ReadableStream and Headers objects. But it’s not about which headers a browser adds.

You a maybe right but that is currently not the priority here. We have chosen to focus on fully implementing the spec. I don’t know enough about Cors and what not the determine what is relevant or not. If you have any specific, constructive and articulated suggestions we are open to discuss them. Even better open a PR.

I wish Node.js’ fetch were freer (“more capable”) than browser’s, while being compatible with browser’s.

This is globalThis.fetch, which has been shipped into Node.js. I want to fully utilize Node.js’ capability using the elegant dependency-less API, fetch.

Nothing fancy is needed; just a few less-restrictive options to customize requests, the spec being the default. So that I don’t need to introduce any unnecessary dependencies (“external packages”) that do the same thing as what Node.js’ fetch does.

I don’t know about Cloudflare Workers, but Deno’s fetch default headers are:

accept: */*
user-agent: Deno/1.20.3
accept-encoding: gzip, br
host: 127.0.0.1:3000

That’s all. That looks fine. Need additional headers? Add them by yourself.

Also look at the headers order.

While the article about sec-fetch-* names “Fetch Metadata Request Headers”, but. It’s not about only Fetch API requests. It’s about every request in browsers, made from https origin: navigate (address bar and link clicks), parser (script/image loading), xhr, fetch.

https://fetch.spec.whatwg.org/#http-network-or-cache-fetch is also written in mind to be for use in browsers.

Using in Node.js sec-fetch-* by default without way to disable it for what? It’s meaningless. Node.js applications don’t have origin, so it’s not possible to say that some request was “cors” (cross-origin), or “same-site”. Node.js applications don’t have navigation bar, global location object, it’s not possible to say what some request was “navigate”.

I like Fetch API (streaming and convenient), but not the browsers limitations (with the corresponding additional “features”). Separate them.

I need a Node.js tool to do network requests with well know and convenient API. I think, people that asked to add fetch to undici also meant that. A Fetch API compatible function.

BTW, response.body of undici is “async iterable” (I can use it in for-await-of), while the native one is not. And it’s okay.

Mimic browsers requests by adding additional headers (sec-fetch-*, sec-ch-ua-*, upgrade-insecure-requests, dnt) (and reordering of them*) is more work for a plugin/other lib that will modify the headers object the appropriate way.

* Wait, I can’t reorder them. It looks that undici just sorts them alphabetically (after host and connection headers). Neither Chrome, not Firefox don’t do that. Both browsers use a custom headers order.

In particular, it’s the addiction case why I need to set "connection": "keep-alive" manually. To reorder the headers.

ModHeader allows to edit and remove entirely any header (except "host", "connection", "cache-control" (on a page reload), "pragma" (on “Disable cache” with DevTools)).

fetch is a browser API.

So, I assume the next things will be implementing of CORS, CORB limitations, restricting of the access to "cookie" header since it’s a browser API? [sarcasm]

I want to use Fetch API because it’s well know and pretty convenient API.

Since it’s a Node.js library I expect that it will have no limitations/additional headers which browsers applies due to security reason. In particular, no sec-fetch-* headers.

Or just add a way to remove the default headers.

With some option for setGlobalDispatcher, for example.

curl http://example.com/ -H 'User-Agent:' -H 'Accept:' -H 'Host:' no one header, but it works (However, with 400 - Bad Request error)

curl https://example.com/ -H 'User-Agent:' -H 'Accept:' only host header and it works with no error.

setGlobalDispatcher(new Agent({ pipelining: 0 }))

It just changes connection: keep-alive to connection: close.

It does not remove any default header.

neither does Chrome.

As I already said, Chrome allows to do it with web extension.

Why it’s not possible to do from a web page context? Because it’s a browser. Web browsers have a lot of limitations.

Browsers do not allow:

• CORS,
• CORB,
• changing/removing “sec-fetch-*”, <–
• reading "cookies" HTTP header
• and other things.

undici is not a browser. Don’t overengineer it.