nginx-proxy-manager: Updating to v2.9.20 breaks cloudflared tunnel

I don’t fully understand all this as I am adjusting Cloudflared IN Cloudflare ONE dash board. I know TLSVerify has to be disabled. I Rolled back to .19 but moving forward what is going to happen here? Will we not be able to SSL all the way or is something going to be adjusted back with NPM or?

What’s the real solution here?

You can adjust these settings on the Cloudflare dashboard.

Go to your tunnel > public hostname > your website.

Then under 'additional application settings > tls, you will see ‘Origin Server Name’. Just put there the hostname of your website. Like example.com or sub.example.com.

Then everything should work with ssl.

Thanks, I see where the issue is. The problem is that cloudflared is not passing the host name it wants to connect to through SNI. One way to fix this would be to add a originServerName with the host name you want to connect to under “ingress” in your cloudflared configuration. Here’s a link to the doc:

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/local-management/ingress/#originservername

I think you should be able to specify “originServerName: *.tech1index.ca” and remove “noTLSVerify: true”, but I’ve never tried doing this with a wildcard certificate. If that doesn’t work, you’d have to replace the wildcard with individual host names.

I’m curious though, Is there a reason why you need to use https at all internally between cloudflared and NPM? It doesn’t really buy you anything and it just wastes CPU because you are decrypting and re-encrypting your web traffic between cloudflared and NPM, which is all within your docker environment. Instead, you could:

1. Disable Force SSL and HTTP/2 support in NPM for your proxy host.
2. Instead of https on port 1443, you would forward to http on port 8080 (or whatever) in your cloudflared configuration.

Traffic will still be encrypted between your clients and the Cloudflare frontend and then across the cloudflare tunnel, but will then be sent plaintext from cloudflared to NPM. This in no way jeopardizes the security of your setup, as the cloudflared-NPM connection is within your docker environment.

On Fri, Mar 17, 2023, at 9:57 AM, Victor Bajada wrote:

@TheBeeZee https://github.com/TheBeeZee

Sure thing, NPM, Cloudflared and my target host are all running on the same Docker host in the same “proxynet” docker network I have created.

Host Application:

• Web server listening on host port 5055 NPM Config is as follows:

• Listens on host port 1433 (mapped to port 443) in container • Proxy Host config points to http://IP_of_Docker_Host:5055 with following options: image https://user-images.githubusercontent.com/50547519/225968787-fa860f69-4a46-44bc-89fd-830decd65509.png

SSL Settings:

image https://user-images.githubusercontent.com/50547519/225968892-27111944-b8d9-4214-a2b4-5e4f75c3645d.png

Advanced:

image https://user-images.githubusercontent.com/50547519/225968968-1cbdba03-5d2d-492e-83ac-377a5b312b3a.png

Cloudflared Config:

image https://user-images.githubusercontent.com/50547519/225969634-9525e549-f24f-44e3-a773-79ef64132109.png

Please let me know if there is any additional info you need.

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2696#issuecomment-1474130354, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R24ACYDSHTVBV7TFU3W4SJZHANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

On 2.9.19 it works by coincidence, not by design. With 2.9.19 the default SSL host is just a normal host, which uses an SSL certificate for a fake domain (localhost.localdomain or some such). If you connect to your NPM host using an invalid name or the IP address, it will present this dummy SSL certificate. By default cloudflare doesn’t specify any hostname, so it lands on this default host. That’s why you had to set “NoVerifyTLS” in the cloudflare configuration, which basically makes the entire connection insecure, completely defeating the purpose of TLS.

With 2.9.20 the configuration was changed so that the default SSL host simply aborts the SSL negotiation if a hostname is not specified during TLS negotiation.

On Sat, Mar 18, 2023, at 7:03 PM, blaine07 wrote:

A wildcard won’t work as nginx won’t know which site you actually want to connect to. The hostname is sent as part of the TLS exchange, before any headers are exchanged, so Cloudflare in your example would need to send the actual hostname to NPM for NPM to know where to route the request to. … https://app.fastmail.com/mail/Inbox/compose/M38892bf5379b568f7e3f79c2?u=b1c14013#

How I have above 100% works on 2.9.19; it quits on 2.9.20+. It forwards any subdomains I forwards to Cloudflared inside CF with the long dns address that points to that instance of Cloudflared.

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2696#issuecomment-1475069904, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R44TYAVUCAXMJOHZYLW4ZSN7ANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

1. Disable Force SSL and HTTP/2 support in NPM for your proxy host. 2. Instead of https on port 1443, you would forward to http on port 8080 (or whatever) in your cloudflared configuration.

Was having the same issue. These steps fixed it for me.

Changed Cloudflared config to not use https. In NPM disabled “Force SSL” and “Http/2 Support”

Restarted the cloudflared docker and everything started working again.

Depending on the rest of the config, you.might also have to specify the name nginx should be using with proxy_ssl_name.

Okay, tried the following:

proxy_ssl_server_name on;
proxy_ssl_name $host;

And that fixed the problem on 2.9.20. Will keep a patched version of 2.9.20 in case this issues arises again though. @tech1ndex Try using $host instead of myurl.com

I was getting this for v2.10.3. I fixed this by updating the cloudflared config. Here is how it looks now. Previously I was using a wildcard rule to catch all subdomains but this no longer works.

credentials-file: /home/nonroot/.cloudflared/xxxxxxxx.json
loglevel: info
ingress:
  - hostname: domain.tld
    service: https://localhost:port
    originRequest:
      noTLSVerify: true
      originServerName: domain.tld
      httpHostHeader: domain.tld
  - hostname: subd.domain.tld
    service: https://localhost:port
    originRequest:
      noTLSVerify: true
      originServerName: subd.domain.tld
      httpHostHeader: subd.domain.tld
   - service: http_status:404

I did not change anything in the npm advanced setting for each proxy host.

Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference.

ingress:
   - hostname: "*.mydomain.com"
     service: https://NGINXPROXYHOSTIP:443
     originRequest:
       originServerName: "anysubdomain.mydomain.com"
       noTLSVerify: true

   - service: http_status:404

…after a long day of dealing with a storm of outages and realizing that NPM’s latest release does this TLS error with CF, this minor, super-small, but very effective tip on the originServerName as a resolution step, is the thing that is gonna let me go to bed at a decent time.
Thank you!

You can remove noTLSVerify.

On Mon, May 1, 2023, at 2:56 AM, Viper wrote:

Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to any of your subdomain (it doesn’t matter which one) on the wildcard hostname tunnel. I don’t have a setup where I could quickly verify that this works, but I don’t see why it shouldn’t.

Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference.

ingress:

• hostname: “*.mydomain.com” service: https://NGINXPROXYHOSTIP:443 originRequest: originServerName: “anysubdomain.mydomain.com” noTLSVerify: true

• service: http_status:404

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2696#issuecomment-1529531088, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5RYN7GFK2V6DPUEAUR3XD6CDNANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

Correct, if you specify each subdomain individually it will work and no need to set noTLSverify. The second solution I suggested above was to set the origin server name to any of your subdomain (it doesn’t matter which one) on the wildcard hostname tunnel. I don’t have a setup where I could quickly verify that this works, but I don’t see why it shouldn’t.

Using a random subdomain did the trick for me. thanks a lot. Here is how my tunnel config looks like for reference.

ingress:
   - hostname: "*.mydomain.com"
     service: https://NGINXPROXYHOSTIP:443
     originRequest:
       originServerName: "anysubdomain.mydomain.com"
       noTLSVerify: true

   - service: http_status:404

There is no change no NPM. The change has to be made to your Cloudflare tunnel configuration.

On Thu, Mar 23, 2023, at 5:32 PM, blaine07 wrote:

Just a quick update here. I was able to setup a test to confirm that setting the origin server name in the TLS settings for the CF tunnel makes things work when the origin server is using TLS with a wildcard certificate. … https://app.fastmail.com/mail/Inbox/compose?u=b1c14013#

So did this change for SURE make 2.9.22 released today - as in I am going to need to go through and make all the changes with/for/now upgrading to 2.9.22?

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2696#issuecomment-1482088533, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5RYTFUTULV33XNCUSK3W5TTS3ANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

Just a quick update here. I was able to setup a test to confirm that setting the origin server name in the TLS settings for the CF tunnel makes things work when the origin server is using TLS with a wildcard certificate.

On Sat, Mar 18, 2023, at 7:39 PM, blaine07 wrote:

You would have to set the origin name on each tunnel host individually, correct. This ensures that CF will set SNI during TLS negotiation. If possible, please test my second suggestion (setting origin name to any of your subdomains on the wildcard), as that would be the quickest and simplest solution. Please report back here if it works or not. … https://app.fastmail.com/mail/Archive/compose/Mff8fda06a08fd987bbf52d98?u=b1c14013#

Earlier I had a ton of issues with .21. Need to make sure my issues were only this and not the database issues discussed with .21 too. I think it was just this the way it was acting and now that I understand. But I did keep getting 503 errors trying to just get to NPM internally too though.

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2696#issuecomment-1475078375, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R3LYUZFD2P5GV4ZHYTW4ZWUZANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

Correct, if you specify each subdomain individually it will work and no need to set noTLSverify.

The second solution I suggested above was to set the origin server name to any of your subdomain (it doesn’t matter which one) on the wildcard hostname tunnel. I don’t have a setup where I could quickly verify that this works, but I don’t see why it shouldn’t.

On Sat, Mar 18, 2023, at 7:25 PM, blaine07 wrote:

On 2.9.19 it works by coincidence, not by design. With 2.9.19 the default SSL host is just a normal host, which uses an SSL certificate for a fake domain (localhost.localdomain or some such). If you connect to your NPM host using an invalid name or the IP address, it will present this dummy SSL certificate. By default cloudflare doesn’t specify any hostname, so it lands on this default host. That’s why you had to set “NoVerifyTLS” in the cloudflare configuration, which basically makes the entire connection insecure, completely defeating the purpose of TLS. With 2.9.20 the configuration was changed so that the default SSL host simply aborts the SSL negotiation if a hostname is not specified during TLS negotiation. … https://app.fastmail.com/mail/Inbox/compose/M9b82ca8e430a9158636e4752?u=b1c14013#

So moving forward, to fix, in cloudflare tunnel in need to remove wildcard and send EACH subdomain in individually? But doing them all individually I wouldn’t need to disable the noTLSVerify? Would I still set origin server name like in my first pic a few posts above this one?

— Reply to this email directly, view it on GitHub https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2696#issuecomment-1475075967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUBW5R7RMYYIFXLFFWY7CGLW4ZVAVANCNFSM6AAAAAAV55G3CY. You are receiving this because you were mentioned.Message ID: @.***>

You might be suffering the same issue I was having earlier, where an option being added - “ssl_reject_handshake” set to on - caused Nginx to reject SSL handshakes to the default handler. Unfortuantly, that blocks services that connects first, then uses the Host header to define where to send their requests. From testing, Apache2 defines who to connect to using SNI and that doesn’t get blocked. Since SNI forwarding from NGINX to NPM appears to be broken at the moment, the fix is sticking to 2.9.19 if using NGINX or a service such as cloudflare.

The commit that introduces this problem: https://github.com/NginxProxyManager/nginx-proxy-manager/commit/a7f0c3b730678ae4352ade2829d891a3ce3cd3bc

@TheBeeZee What was the rationale behind adding that option to the default config? Couldn’t having that behind a togglable option been better, so frontends like NGINX standalone or cloudflare don’t break?