shellhub: gateway: reports "Broken header" when using nginx ssl reverse proxy

Shellhub v0.3.7 with nginx/1.14.2

I have a self-hosted Shellhub server started with the standard docker-compose method, listening on external port 8090, and it works fine. I’ve set up an ssl nginx reverse proxy (not dockerized) in front of the ShellHub server listening on port 8095, passing data to the Shellhub gateway on port 8090. HTTPS access via the proxy to the Shellhub server works fine with SHELLHUB_PROXY=false, showing that the proxy works for HTTP.

After setting SHELLHUB_PROXY=true and restarting Shellhub with docker-compose, every access via the proxy fails with a “broken header” error. An example is:

gateway_1_f5da0b62b647 | Host: localhost:8090
gateway_1_f5da0b62b647 | Connection: close
gateway_1_f5da0b62b647 | User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0
gateway_1_f5da0b62b647 | Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
gateway_1_f5da0b62b647 | Accept-Language: en-GB,en;q=0.5
gateway_1_f5da0b62b647 | Accept-Encoding: gzip, deflate, br
gateway_1_f5da0b62b647 | DNT: 1
gateway_1_f5da0b62b647 | Upgrade-Insecure-Requests: 1
gateway_1_f5da0b62b647 | Cache-Control: max-age=0
gateway_1_f5da0b62b647 |
gateway_1_f5da0b62b647 | " while reading PROXY protocol, client: 172.22.0.1, server: 0.0.0.0:80

[The example above has been edited: the first example posted was from a Shellhub client accessing the server without going via the proxy. The edited example is from an attempt to access the web interface from a standard web client through the proxy.]

My nginx reverse proxy definition is:

server {
  listen        8095 ssl;

  ssl_certificate     /etc/letsencrypt/live/domain-name/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/domain-name/privkey.pem;

  location / {
    proxy_pass  http://localhost:8090;
  }
}

The Shellhub .env file is:

SHELLHUB_VERSION=v0.3.7
SHELLHUB_HTTP_PORT=8090
SHELLHUB_SSH_PORT=9822
SHELLHUB_PROXY=true
SHELLHUB_HOSTED=false

Also reported by @jcampos79_gitlab on the Gitter channel.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 27 (25 by maintainers)

Commits related to this issue

agent: fix 'index out of range' error in logging This alsos move log entry to a location where connection is fully established. See @sixhills comment: https://github.com/shellhub-io/shellhub/issues/... — committed to shellhub-io/shellhub by gustavosbarreto 4 years ago
agent: fix 'index out of range' error in logging This alsos move log entry to a location where connection is fully established. See @sixhills comment: https://github.com/shellhub-io/shellhub/issues/... — committed to shellhub-io/shellhub by gustavosbarreto 4 years ago
agent: fix 'index out of range' error in logging This alsos move log entry to a location where connection is fully established. See @sixhills comment: https://github.com/shellhub-io/shellhub/issues/... — committed to shellhub-io/shellhub by gustavosbarreto 4 years ago

Most upvoted comments

Here is a step by step how to achieve a SSL proxy in-front of ShellHub using docker-nginx-auto-ssl container:

Create a docker-compose.override.yml:

version: '3.7'

services:
  nginx:
    image: valian/docker-nginx-auto-ssl:1.2.0
    restart: on-failure
    ports:
      - 80:80
      - 443:443
    environment:
      ALLOWED_DOMAINS: '${SHELLHUB_DOMAIN}'
      SITES: '${SHELLHUB_DOMAIN}=gateway:80'

Unexpose gateway port on host network by setting SHELLHUB_HTTP_PORT=0 in .env file.
Set SHELLHUB_DOMAIN for desired domain in .env file. e.g. SHELLHUB_DOMAIN=example.com

What do you think about adding this step by step to documentation?

gustavosbarreto on Aug 28, 2020

@gustavosbarreto Thanks for fixing this. The revised code is much better than mine.

sixhills on Aug 31, 2020

There may be a timing issue. Sometimes, the valian nginx container fails to start with:

2020/08/28 15:00:26 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/conf.d/fqdn.conf:18

Adding “restart: on-failure” to the nginx definition fixes this, although there may be a better way.

sixhills on Aug 28, 2020

It works! I just had to insert:

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

at the start of the file.

I thought there was something wrong with my nginx config but wasn’t sure what it was. It would be very helpful to publish that example conf file in the “Production Deployment” section of the manual.

Thanks again for the excellent support!

sixhills on Aug 27, 2020

Thanks for the advice. I’ve made that change but nginx then fails to start:

nginx[3322]: nginx: [emerg] "proxy_protocol" directive is not allowed here in /etc/nginx/sites-enabled/shellhub.conf:3

To use ‘proxy_protocol’ you need to use a stream server instead of http server.

Does this works for you?

/etc/nginx/nginx.conf:

worker_processes  1;

events {
    worker_connections  1024;
}

stream {
    upstream shellhub {
        least_conn;
        server localhost:8090;
    }

    server {
        listen 8095 ssl;
        proxy_protocol on;

        ssl_certificate     /etc/letsencrypt/live/domain-name/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/domain-name/privkey.pem;

        proxy_pass  shellhub;
    }
}

gustavosbarreto on Aug 27, 2020