rancher: AWS ELB + Rancher HA Returning HTTP instead of HTTPS
Rancher Version: Server v1.0.1
OS: RancherOS v0.4.4
Expected: HTTPS ELB SSL to HTTP Rancher TCP PP returns HTTPS endpoint to client/end-user.
Results: Rancher Server returns HTTP endpoint in almost every scenario.
Get ready for a long post. For the past few days, I have been troubleshooting this issue and figured it was time to open a ticket.
When using HTTPS only behind an ELB with PP, the Rancher Server is returning an HTTP API endpoint instead of HTTPS, this causes the JS to go into a constant loop.
Using cURL, I decided to go through every permutation I could think of and paste the results here to see if this is a bug or I am doing something wrong, enjoy.
Permutations and Results:
(waited at least 60 secs between each change for ELBs to cycle if needed)
Created Fresh ELB
ELB 443 SSL to Rancher HA 80 TCP
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 80 TCP" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 80 TCP
X-API-Schemas: http://rancher.example.com/v1/schemas
X-API-Client-IP: 10.0.103.184
[~/Code]
Result: PASS
- HTTP Schema address returned, as expected.
- Internal AWS Client IP returned, as expected.
ELB 443 SSL to Rancher HA 81 TCP (No PP)
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 81 TCP (No PP)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 81 TCP
[~/Code]
Result: PASS
- Nothing returned from server, as expected.
ELB 443 SSL to Rancher HA 443 TCP
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 443 TCP" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 443 TCP
[~/Code]
Result: PASS
- Nothing returned from server, as expected.
ELB 443 SSL to Rancher HA 443 SSL
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 443 SSL" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 443 SSL
X-API-Schemas: https://rancher.example.com/v1/schemas
X-API-Client-IP: 10.0.103.184
Result: PASS
- HTTPS Schema address returned, as expected.
- Internal AWS Client IP returned, as expected.
ELB 443 SSL to Rancher HA 444 TCP (No PP)
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 444 TCP (No PP)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 444 TCP (No PP)
[~/Code]
Result: PASS
- Nothing returned from server, as expected.
Created PP Policy:
[~/Code] aws elb create-load-balancer-policy --load-balancer-name rancher-masters-elb --policy-name rancher-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true --profile prod
[~/Code]
ELB 443 SSL to Rancher HA 81 TCP (PP Enabled)
[~/Code] aws elb set-load-balancer-policies-for-backend-server --load-balancer-name rancher-masters-elb --instance-port 81 --policy-names rancher-ProxyProtocol-policy --profile prod
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 81 TCP (PP Enabled)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 81 TCP (PP Enabled)
X-API-Schemas: http://rancher.example.com/v1/schemas
X-API-Client-IP: 173.16.1.105
[~/Code]
Result: FAIL
- HTTP Schema address returned. Expected HTTPS Schema address.
- Actual client IP returned, as expected with PP Enabled.
ELB 443 SSL to Rancher HA 444 TCP (PP Enabled)
[~/Code] aws elb set-load-balancer-policies-for-backend-server --load-balancer-name rancher-masters-elb --instance-port 444 --policy-names rancher-ProxyProtocol-policy --profile prod
[~/Code] sleep 60 && echo "ELB 443 SSL to Rancher HA 444 TCP (PP Enabled)" && curl -is https://rancher.example.com --cookie PL=rancher --cookie CSRF=AAB761C194939684FD197467018747CD --cookie token=F2ZsRGjk9uUmahZXnbJj5c79qpyTPikdA7PhYrhu | grep -E (X-API-Schemas|X-API-Client-IP)
ELB 443 SSL to Rancher HA 444 TCP (PP Enabled)
X-API-Schemas: http://rancher.example.com/v1/schemas
X-API-Client-IP: 173.16.1.105
[~/Code]
Result: FAIL
- HTTP Schema address returned. Expected HTTPS Schema address.
- Actual client IP returned, as expected with PP Enabled.
Other Configurations
Cattle HA Settings:
CATTLE_HA_HOST_REGISTRATION_URL=https://rancher.example.com
CATTLE_HA_PORT_HTTP=80
CATTLE_HA_PORT_HTTPS=443
CATTLE_HA_PORT_PP_HTTP=81
CATTLE_HA_PORT_PP_HTTPS=444
Load Balancer Config
[~/Code] aws elb describe-load-balancers --load-balancer-name rancher-masters-elb --profile prod
{
"LoadBalancerDescriptions": [
{
"Subnets": [
"subnet-4be08123",
"subnet-a8e0b123",
"subnet-d8237123"
],
"CanonicalHostedZoneNameID": "Z3DZXE0Q79N123",
"CanonicalHostedZoneName": "rancher-masters-elb-1692841234.us-east-1.elb.amazonaws.com",
"ListenerDescriptions": [
{
"Listener": {
"InstancePort": 81,
"SSLCertificateId": "arn:aws:acm:us-east-1:213321824123:certificate/9f0907a7-096c-4537-a257-d32294955bd6",
"LoadBalancerPort": 443,
"Protocol": "SSL",
"InstanceProtocol": "TCP"
},
"PolicyNames": [
"ELBSecurityPolicy-2015-05"
]
}
],
"HealthCheck": {
"HealthyThreshold": 2,
"Interval": 30,
"Target": "TCP:80",
"Timeout": 3,
"UnhealthyThreshold": 2
},
"VPCId": "vpc-55d32032",
"BackendServerDescriptions": [
{
"InstancePort": 81,
"PolicyNames": [
"rancher-ProxyProtocol-policy"
]
},
{
"InstancePort": 444,
"PolicyNames": [
"rancher-ProxyProtocol-policy"
]
}
],
"Instances": [
{
"InstanceId": "i-6dfe1123"
},
{
"InstanceId": "i-7c335123"
},
{
"InstanceId": "i-a7156123"
}
],
"DNSName": "rancher-masters-elb-1692841234.us-east-1.elb.amazonaws.com",
"SecurityGroups": [
"sg-3cdd2123"
],
"Policies": {
"LBCookieStickinessPolicies": [],
"AppCookieStickinessPolicies": [],
"OtherPolicies": [
"rancher-ProxyProtocol-policy",
"ELBSecurityPolicy-2015-05"
]
},
"LoadBalancerName": "rancher-masters-elb",
"CreatedTime": "2016-05-05T13:10:24.020Z",
"AvailabilityZones": [
"us-east-1b",
"us-east-1c",
"us-east-1e"
],
"Scheme": "internet-facing",
"SourceSecurityGroup": {
"OwnerAlias": "213321824123",
"GroupName": "rancher-masters-elb"
}
}
]
}
[~/Code]
Other: Spoke with @vincent99 & @ibuildthecloud about this on IRC, so, tagging them here in case they have insight or reference.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 17 (3 by maintainers)
Working Setup for me - SSL Termination on ELB: aws elb create-load-balancer-policy --load-balancer-name rancher-management-external --policy-name ProxyProtocol --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
Configure Listener: SSL (Secure TCP) 443 -> TCP 81
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name rancher-management-external --instance-port 81 --policy-names ProxyProtocol
Working Setup - SSL between ELB/Rancher
aws elb create-load-balancer-policy --load-balancer-name rancher-management-external --policy-name ProxyProtocol --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
Configure Listener: SSL (Secure TCP) 443 -> TCP 444
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name rancher-management-external --instance-port 444 --policy-names ProxyProtocol
Thanks for help everyone involved 👍
Yeah, it’s slated for both both releases.
Ping @LLParse. Aren’t you looking into this now?