Rancher Version: Server v1.0.1

OS: RancherOS v0.4.4

Expected: HTTPS ELB SSL to HTTP Rancher TCP PP returns HTTPS endpoint to client/end-user.

Results: Rancher Server returns HTTP endpoint in almost every scenario.

Get ready for a long post. For the past few days, I have been troubleshooting this issue and figured it was time to open a ticket.

When using HTTPS only behind an ELB with PP, the Rancher Server is returning an HTTP API endpoint instead of HTTPS, this causes the JS to go into a constant loop.

Using cURL, I decided to go through every permutation I could think of and paste the results here to see if this is a bug or I am doing something wrong, enjoy.

Permutations and Results:

(waited at least 60 secs between each change for ELBs to cycle if needed)

Created Fresh ELB

ELB 443 SSL to Rancher HA 80 TCP

[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 80 TCP" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 80 TCP
X-API-Schemas: http://rancher.example.com/v1/schemas
X-API-Client-IP: 10.0.103.184
[~/Code]

Result: PASS

• HTTP Schema address returned, as expected.
• Internal AWS Client IP returned, as expected.

ELB 443 SSL to Rancher HA 81 TCP (No PP)

[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 81 TCP (No PP)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 81 TCP
[~/Code]

Result: PASS

• Nothing returned from server, as expected.

ELB 443 SSL to Rancher HA 443 TCP

[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 443 TCP" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 443 TCP
[~/Code]

Result: PASS

• Nothing returned from server, as expected.

ELB 443 SSL to Rancher HA 443 SSL

[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 443 SSL" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 443 SSL
X-API-Schemas: https://rancher.example.com/v1/schemas
X-API-Client-IP: 10.0.103.184

Result: PASS

• HTTPS Schema address returned, as expected.
• Internal AWS Client IP returned, as expected.

ELB 443 SSL to Rancher HA 444 TCP (No PP)

[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 444 TCP (No PP)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 444 TCP (No PP)
[~/Code]

Result: PASS

• Nothing returned from server, as expected.

Created PP Policy:

[~/Code] aws elb create-load-balancer-policy --load-balancer-name rancher-masters-elb --policy-name rancher-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true --profile prod
[~/Code]

ELB 443 SSL to Rancher HA 81 TCP (PP Enabled)

[~/Code] aws elb set-load-balancer-policies-for-backend-server --load-balancer-name rancher-masters-elb --instance-port 81 --policy-names rancher-ProxyProtocol-policy --profile prod
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 81 TCP (PP Enabled)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 81 TCP (PP Enabled)
X-API-Schemas: http://rancher.example.com/v1/schemas
X-API-Client-IP: 173.16.1.105
[~/Code]

Result: FAIL

• HTTP Schema address returned. Expected HTTPS Schema address.
• Actual client IP returned, as expected with PP Enabled.

ELB 443 SSL to Rancher HA 444 TCP (PP Enabled)

[~/Code] aws elb set-load-balancer-policies-for-backend-server --load-balancer-name rancher-masters-elb --instance-port 444 --policy-names rancher-ProxyProtocol-policy --profile prod
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 444 TCP (PP Enabled)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 444 TCP (PP Enabled)
X-API-Schemas: http://rancher.example.com/v1/schemas
X-API-Client-IP: 173.16.1.105
[~/Code]

Result: FAIL

• HTTP Schema address returned. Expected HTTPS Schema address.
• Actual client IP returned, as expected with PP Enabled.

Other Configurations

Cattle HA Settings:

CATTLE_HA_HOST_REGISTRATION_URL=https://rancher.example.com
CATTLE_HA_PORT_HTTP=80
CATTLE_HA_PORT_HTTPS=443
CATTLE_HA_PORT_PP_HTTP=81
CATTLE_HA_PORT_PP_HTTPS=444

Load Balancer Config

[~/Code] aws elb describe-load-balancers --load-balancer-name rancher-masters-elb --profile prod
{
    "LoadBalancerDescriptions": [
        {
            "Subnets": [
                "subnet-4be08123",
                "subnet-a8e0b123",
                "subnet-d8237123"
            ],
            "CanonicalHostedZoneNameID": "Z3DZXE0Q79N123",
            "CanonicalHostedZoneName": "rancher-masters-elb-1692841234.us-east-1.elb.amazonaws.com",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "InstancePort": 81,
                        "SSLCertificateId": "arn:aws:acm:us-east-1:213321824123:certificate/9f0907a7-096c-4537-a257-d32294955bd6",
                        "LoadBalancerPort": 443,
                        "Protocol": "SSL",
                        "InstanceProtocol": "TCP"
                    },
                    "PolicyNames": [
                        "ELBSecurityPolicy-2015-05"
                    ]
                }
            ],
            "HealthCheck": {
                "HealthyThreshold": 2,
                "Interval": 30,
                "Target": "TCP:80",
                "Timeout": 3,
                "UnhealthyThreshold": 2
            },
            "VPCId": "vpc-55d32032",
            "BackendServerDescriptions": [
                {
                    "InstancePort": 81,
                    "PolicyNames": [
                        "rancher-ProxyProtocol-policy"
                    ]
                },
                {
                    "InstancePort": 444,
                    "PolicyNames": [
                        "rancher-ProxyProtocol-policy"
                    ]
                }
            ],
            "Instances": [
                {
                    "InstanceId": "i-6dfe1123"
                },
                {
                    "InstanceId": "i-7c335123"
                },
                {
                    "InstanceId": "i-a7156123"
                }
            ],
            "DNSName": "rancher-masters-elb-1692841234.us-east-1.elb.amazonaws.com",
            "SecurityGroups": [
                "sg-3cdd2123"
            ],
            "Policies": {
                "LBCookieStickinessPolicies": [],
                "AppCookieStickinessPolicies": [],
                "OtherPolicies": [
                    "rancher-ProxyProtocol-policy",
                    "ELBSecurityPolicy-2015-05"
                ]
            },
            "LoadBalancerName": "rancher-masters-elb",
            "CreatedTime": "2016-05-05T13:10:24.020Z",
            "AvailabilityZones": [
                "us-east-1b",
                "us-east-1c",
                "us-east-1e"
            ],
            "Scheme": "internet-facing",
            "SourceSecurityGroup": {
                "OwnerAlias": "213321824123",
                "GroupName": "rancher-masters-elb"
            }
        }
    ]
}
[~/Code]

Other: Spoke with @vincent99 & @ibuildthecloud about this on IRC, so, tagging them here in case they have insight or reference.