request: Can't pipe from request that has a POST body

I know this is old, but I had a hard time finding a suitable solution to this, and thought it might be useful to others.

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post({ url: 'http://www.example.com', form: { foo: 'bar' }}), {end: false}).pipe(res);
});

app.listen(process.env.PORT || 3000);

As shown in the https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options, adding this makes it so the new stream isn’t automatically terminated when req is terminated.

While this caused this example to work, I’ve noticed some issues making a proxy for other methods using this. The body somehow causes it to close everything properly, but without a body it still hangs. If using a json body parser, this shouldn’t cause a problem since it causes the body to be read as {} even when there is no body, but without one it could still have issues. My current solution is to check for a content-length header on the request, and use the options object if one exists.

I actually used .pipe(res) in my real example and I verified that it doesn’t work. It doesn’t appear to send the request and it hangs. Since your example works, I quickly found the cause of the problem:

app.use(bodyParser.urlencoded({ extended: true }));

It looks to me like when I use body-parser, the piping no longer works. Do you agree and, if so, do you have any idea what the cause might be?

Yep, if you take a look at my first comment again:

// put it before any other middleware

body-parser processes the request before entering your middleware, but you need the raw request to pipe it successfully.

I don’t know too much about express or request (just using these for a single-page app proxy without really understanding how they work), but this fixed the write after end error for me:

request.post({ url: 'http://www.example.com', form: { foo: 'bar' }}).pipe(res);

Instead of:

req.pipe(request.post({ url: 'http://www.example.com', form: { foo: 'bar' }})).pipe(res);

GET requests does not have a body, that’s by design.

This one works (your example):

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post('http://www.example.com'));
});

app.listen(process.env.PORT || 3000);

Using this to make the request:

var request = require('request')
request.post('http://localhost:3000', {form:{some:'data'}})

some=data got sent.

You’ll see that the body is undefined in the new object.

Where is that, I’m not following you.

Let me explain my use case. I implemented a simple web API proxy (see https://github.com/salsita/web-api-proxy). All it does is receive HTTP requests and redirect them to another host, replacing URL parameters if necessary with environment variables on the proxy server. The point is to avoid having to embed secret keys into client code.

It works great for GET requests, but I recently had a requirement to use it for POST as well. So I’m not initializing the body of my request as in your previous comment since the body is coming through the Express request. Your most recent example definitely works but that doesn’t really help me. What I need to do is:

req.pipe(request.post('http://some/new/url'));

It seems to me that if req has a body, it doesn’t get copied over in this case. And anyway, the truth is that I do need to modify the body (since it might have placeholders that I need to replace using environment variables) so I’m going to have to do:

req.pipe(request.post({ url: 'http://some/new/url', form: { some: 'form', data: 'here' }));

That causes an exception to be thrown due to what I suspect is a bug in request.

Anyway, I can build a completely new request object myself, but I wanted to make sure that I wasn’t missing something and also to report this issue in case it is a bug.

@simov It’s the sample in my original post. If I use this one then the body is not sent:

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post('http://www.example.com'));
});

app.listen(process.env.PORT || 3000);

If I use this then I get that exception:

'use strict';

let express = require('express');
let request = require('request');

let app = express();

app.use('/', (req, res) => {
  req.pipe(request.post({ url: 'http://www.example.com', form: { foo: 'bar' }}));
});

app.listen(process.env.PORT || 3000);

Try with this one:

var express = require('express')
var request = require('request')

var app = express()

// put it before any other middleware
app.use(function(req, res) {
  req.pipe(
    request[req.method.toLowerCase()]('http://example.com'))
  .pipe(res)
})

app.listen(3000, function () {
  console.log('Express server listening on port 3000')
})

In case you want to modify the body, just construct a new request and pipe to the response.