sails: sails.js /socket.io/ URLs behind nginx leads to lots of 'upstream timed out'
I’m running several sails.js instances behind an nginx proxy with sticky sessions. And I keep seeing a lot of messages in my nginx error.log regarding sails.js /socket.io/ URLs timing out:
2016/01/04 20:55:15 [error] 12106#12106: *402088 upstream timed out (110: Connection timed out) while reading response header from upstream, client: x.x.x.x, server: example.com, request: "GET /socket.io/?__sails_io_sdk_version=0.11.0&__sails_io_sdk_platform=browser&__sails_io_sdk_language=javascript&EIO=3&transport=polling&t=1451930055065-4&sid=jvekCYDAcFfu0PLdAAL6 HTTP/1.1", upstream: "http://127.0.0.1:3001/socket.io/?__sails_io_sdk_version=0.11.0&__sails_io_sdk_platform=browser&__sails_io_sdk_language=javascript&EIO=3&transport=polling&t=1451930055065-4&sid=jvekCYDAcFfu0PLdAAL6", host: "example.com", referrer: "https://example.com/languageExchange/chat/63934"
2016/01/04 20:55:17 [error] 12105#12105: *402482 upstream prematurely closed connection while reading response header from upstream, client: y.y.y.y, server: example.com, request: "GET /socket.io/?__sails_io_sdk_version=0.11.0&__sails_io_sdk_platform=browser&__sails_io_sdk_language=javascript&EIO=3&transport=websocket&sid=QnAe1jiKEHgj-zlKAAKu HTTP/1.1", upstream: "http://127.0.0.1:3001/socket.io/?__sails_io_sdk_version=0.11.0&__sails_io_sdk_platform=browser&__sails_io_sdk_language=javascript&EIO=3&transport=websocket&sid=QnAe1jiKEHgj-zlKAAKu", host: "example.com"
It doesn’t happen for every client, but the number of such messages is significant. And sails.js does not show any relevant errors.
How should I investigate the nature of these issues?
Here’s what I’ve tried so far (and it didn’t help):
- Upgrade socket.io client to the latest version so far (1.3.7)
- Explicitly turn off caching for /socket.io/ requests in nginx
Here’s the relevant config files:
- sails sockets.js:
adapter: 'socket.io-redis'
- nginx:
location ^~ /socket.io/ {
proxy_pass http://sails;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_no_cache true;
proxy_cache_bypass true;
proxy_redirect off;
proxy_intercept_errors off;
}
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 17 (3 by maintainers)
I read that this may be due to sticky sessions issue. Nginx (free) does not support sticky sessions, therefore long polling will result in a time out error. If you are using Nginx Plus then you can enable sticky sessions and the issue will not be apparent. As suggested by @mikermcneil I suspect dropping the polling and going for basic websocket support when using free nginx will alleviate the issue. I will make the changes to my server config tonight and report back to see if this alleviates the issue.
So the issue is a bad configuration on the NGINX sites-avalable conf file.
should be
Pretty basic stuff: the “location” is not forwarded to the proxy_pass (why would it be, right?) – So you need to make sure the socket requests are redirected to the exact socket endpoint.