rancher: Rancher Server 1.4.0 unable to make calls to api (getsockopt: connection refused)

Rancher Versions: Server: v1.4.0

Docker Version: Docker version 1.12.6, build 7392c3b/1.12.6

OS and where are the hosts located? (cloud, bare metal, etc): Amazon Linux (AWS)

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) Single node, external DB (RDS)

Environment Type: (Cattle/Kubernetes/Swarm/Mesos) Cattle

Steps to Reproduce:

docker run -d --restart=always --name=rancher-server --expose 8080/tcp \
           -e CATTLE_DB_CATTLE_MYSQL_HOST=<my_db_host> \
           -e CATTLE_DB_CATTLE_MYSQL_PORT=3306 \
           -e CATTLE_DB_CATTLE_MYSQL_NAME=<my_db_name> \
           -e CATTLE_DB_CATTLE_USERNAME=<my_db_user> \
           -e CATTLE_DB_CATTLE_PASSWORD=<my_db_password> \
           -e VIRTUAL_HOST=<my_hostname> \
           -e VIRTUAL_PORT=8080 \
              rancher/server:v1.4.0

Results: every request to localhost:8080 fails with something along the lines of Unable to create event router" error="Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused

# output of docker logs rancher-server
# requests to localhost:8080 fail (see below)
CATTLE_AGENT_PACKAGE_HOST_API_URL=/usr/share/cattle/artifacts/host-api.tar.gz
CATTLE_AGENT_PACKAGE_PYTHON_AGENT_URL=/usr/share/cattle/artifacts/go-agent.tar.gz
CATTLE_API_UI_URL=//releases.rancher.com/api-ui/1.0.8
CATTLE_CATTLE_VERSION=v0.176.8
CATTLE_DB_CATTLE_DATABASE=mysql
CATTLE_DB_CATTLE_MYSQL_HOST=<my_db_host>
CATTLE_DB_CATTLE_MYSQL_NAME=<my_db_name>
CATTLE_DB_CATTLE_MYSQL_PORT=3306
CATTLE_DB_CATTLE_USERNAME=<my_db_user>
CATTLE_GRAPHITE_HOST=
CATTLE_GRAPHITE_PORT=
CATTLE_HOME=/var/lib/cattle
CATTLE_HOST_API_PROXY_MODE=embedded
CATTLE_LOGBACK_OUTPUT_GELF_HOST=
CATTLE_LOGBACK_OUTPUT_GELF_PORT=
CATTLE_RANCHER_CLI_VERSION=v0.4.1
CATTLE_RANCHER_COMPOSE_VERSION=v0.12.2
CATTLE_RANCHER_SERVER_IMAGE=rancher/server
CATTLE_RANCHER_SERVER_VERSION=v1.4.0
CATTLE_USE_LOCAL_ARTIFACTS=true
DEFAULT_CATTLE_API_UI_CSS_URL=/api-ui/ui.min.css
DEFAULT_CATTLE_API_UI_INDEX=//releases.rancher.com/ui/1.4.6
DEFAULT_CATTLE_API_UI_JS_URL=/api-ui/ui.min.js
DEFAULT_CATTLE_AUTH_SERVICE_EXECUTE=true
DEFAULT_CATTLE_CATALOG_EXECUTE=true
DEFAULT_CATTLE_CATALOG_URL=community=https://git.rancher.io/community-catalog.git,library=https://git.rancher.io/rancher-catalog.git
DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE=true
DEFAULT_CATTLE_MACHINE_EXECUTE=true
DEFAULT_CATTLE_RANCHER_CLI_DARWIN_URL=https://releases.rancher.com/cli/v0.4.1/rancher-darwin-amd64-v0.4.1.tar.gz
DEFAULT_CATTLE_RANCHER_CLI_LINUX_URL=https://releases.rancher.com/cli/v0.4.1/rancher-linux-amd64-v0.4.1.tar.gz
DEFAULT_CATTLE_RANCHER_CLI_WINDOWS_URL=https://releases.rancher.com/cli/v0.4.1/rancher-windows-386-v0.4.1.zip
DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-darwin-amd64-v0.12.2.tar.gz
DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-linux-amd64-v0.12.2.tar.gz
DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-windows-386-v0.12.2.zip
DEFAULT_CATTLE_SECRETS_API_EXECUTE=true
DEFAULT_CATTLE_WEBHOOK_SERVICE_EXECUTE=true
21:39:29.194 [main] INFO  ConsoleStatus - Loading configuration
2017-02-07 21:39:37,309 INFO    [main] [ConsoleStatus] Starting DB migration 
2017-02-07 21:39:40,805 INFO    [main] [ConsoleStatus] DB migration done 
2017-02-07 21:39:41,492 INFO    [main] [ConsoleStatus] Cluster membership changed [127.0.0.1:9345] 
2017-02-07 21:39:41,495 INFO    [main] [ConsoleStatus] Checking cluster state on start-up 
2017-02-07 21:39:41,495 INFO    [main] [ConsoleStatus] Waiting to become master 
2017-02-07 21:40:05,058 INFO    [main] [ConsoleStatus] Loading processes 
2017-02-07 21:40:05,772 INFO    [main] [ConsoleStatus] Starting [1/85]: LockDelegatorImpl 
2017-02-07 21:40:05,777 INFO    [main] [ConsoleStatus] Starting [2/85]: AnnotatedListenerRegistration 
2017-02-07 21:40:05,797 INFO    [main] [ConsoleStatus] Starting [3/85]: EventService 
2017-02-07 21:40:05,797 INFO    [main] [ConsoleStatus] Starting [4/85]: DefaultObjectMetaDataManager 
2017-02-07 21:40:07,486 INFO    [main] [ConsoleStatus] Starting [5/85]: JsonDefaultsProvider 
2017-02-07 21:40:07,590 INFO    [main] [ConsoleStatus] Starting [6/85]: ObjectDefaultsPostInstantiationHandler 
2017-02-07 21:40:07,590 INFO    [main] [ConsoleStatus] Starting [7/85]: DefaultProcessManager 
2017-02-07 21:40:07,593 INFO    [main] [ConsoleStatus] Starting [8/85]: SampleDataStartupV3 
2017-02-07 21:40:07,607 INFO    [main] [ConsoleStatus] Starting [9/85]: TaskManagerImpl 
2017-02-07 21:40:07,787 INFO    [main] [ConsoleStatus] Starting [10/85]: ServiceAccountCreateStartup 
2017-02-07 21:40:07,818 INFO    [main] [ConsoleStatus] Starting [11/85]: SchemaFactory:v1-admin 
2017-02-07 21:40:08,099 INFO    [main] [ConsoleStatus] Starting [12/85]: SchemaFactory:v1-agent 
2017-02-07 21:40:08,105 INFO    [main] [ConsoleStatus] Starting [13/85]: SchemaFactory:v1-agentRegister 
2017-02-07 21:40:08,112 INFO    [main] [ConsoleStatus] Starting [14/85]: SchemaFactory:v1-base 
2017-02-07 21:40:08,381 INFO    [main] [ConsoleStatus] Starting [15/85]: SchemaFactory:v1-member 
2017-02-07 21:40:08,517 INFO    [main] [ConsoleStatus] Starting [16/85]: SchemaFactory:v1-owner 
2017-02-07 21:40:08,678 INFO    [main] [ConsoleStatus] Starting [17/85]: SchemaFactory:v1-project 
2017-02-07 21:40:08,802 INFO    [main] [ConsoleStatus] Starting [18/85]: SchemaFactory:v1-projectadmin 
2017-02-07 21:40:08,920 INFO    [main] [ConsoleStatus] Starting [19/85]: SchemaFactory:v1-readAdmin 
2017-02-07 21:40:09,044 INFO    [main] [ConsoleStatus] Starting [20/85]: SchemaFactory:v1-readonly 
2017-02-07 21:40:09,108 INFO    [main] [ConsoleStatus] Starting [21/85]: SchemaFactory:v1-register 
2017-02-07 21:40:09,116 INFO    [main] [ConsoleStatus] Starting [22/85]: SchemaFactory:v1-restricted 
2017-02-07 21:40:09,218 INFO    [main] [ConsoleStatus] Starting [23/85]: SchemaFactory:v1-service 
2017-02-07 21:40:09,285 INFO    [main] [ConsoleStatus] Starting [24/85]: SchemaFactory:v1-token 
2017-02-07 21:40:09,289 INFO    [main] [ConsoleStatus] Starting [25/85]: SchemaFactory:v1-user 
2017-02-07 21:40:09,354 INFO    [main] [ConsoleStatus] Starting [26/85]: AgentBasedProcessHandler 
2017-02-07 21:40:09,354 INFO    [main] [ConsoleStatus] Starting [27/85]: AgentHostStateUpdate 
2017-02-07 21:40:09,355 INFO    [main] [ConsoleStatus] Starting [28/85]: AuthServiceLauncher 
2017-02-07 21:40:09,365 INFO    [main] [ConsoleStatus] Starting [29/85]: BackupCreate 
2017-02-07 21:40:09,367 INFO    [main] [ConsoleStatus] Starting [30/85]: BackupRemove 
2017-02-07 21:40:09,367 INFO    [main] [ConsoleStatus] Starting [31/85]: CatalogLauncher 
2017-02-07 21:40:09,367 INFO    [main] [ConsoleStatus] Starting [32/85]: ComposeExecutorLauncher 
2017-02-07 21:40:09,372 INFO    [main] [ConsoleStatus] Starting [33/85]: ConfigItemRegistryImpl 
2017-02-07 21:40:09,562 INFO    [main] [ConsoleStatus] Starting [34/85]: ConfigItemServerImpl 
2017-02-07 21:40:09,884 INFO    [main] [ConsoleStatus] Starting [35/85]: ConfigUpdatePublisher 
2017-02-07 21:40:09,884 INFO    [main] [ConsoleStatus] Starting [36/85]: DataManager 
2017-02-07 21:40:09,884 INFO    [main] [ConsoleStatus] Starting [37/85]: DefaultAuthorizationProvider 
time="2017-02-07T21:40:11Z" level=info msg="Starting rancher-compose-executor" version=v0.12.2 
time="2017-02-07T21:40:11Z" level=fatal msg="Unable to create event router" error="Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused" 
time="2017-02-07T21:40:11Z" level=info msg="Starting Rancher Catalog service" 
time="2017-02-07T21:40:11Z" level=info msg="Using catalog community=https://git.rancher.io/community-catalog.git" 
time="2017-02-07T21:40:11Z" level=info msg="Using catalog library=https://git.rancher.io/rancher-catalog.git" 
time="2017-02-07T21:40:11Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v1: dial tcp [::1]:8080: getsockopt: connection refused" 
2017-02-07 21:40:12,647 INFO    [main] [ConsoleStatus] Starting [38/85]: DefaultJooqResourceManager 
2017-02-07 21:40:12,648 INFO    [main] [ConsoleStatus] Starting [39/85]: DefaultObjectSerializerFactoryImpl 
2017-02-07 21:40:12,648 INFO    [main] [ConsoleStatus] Starting [40/85]: DockerComposeStackHandler 
2017-02-07 21:40:12,649 INFO    [main] [ConsoleStatus] Starting [41/85]: DynamicExtensionManager 
2017-02-07 21:40:12,649 INFO    [main] [ConsoleStatus] Starting [42/85]: ExtensionResourceManagerLocator 
2017-02-07 21:40:12,649 INFO    [main] [ConsoleStatus] Starting [43/85]: HostApiRSAKeyProvider 
2017-02-07 21:40:12,652 INFO    [main] [ConsoleStatus] Starting [44/85]: ImageStoragePoolMapActivate 
2017-02-07 21:40:12,652 INFO    [main] [ConsoleStatus] Starting [45/85]: InstanceHostMapActivate 
2017-02-07 21:40:12,653 INFO    [main] [ConsoleStatus] Starting [46/85]: InstanceHostMapDeactivate 
2017-02-07 21:40:12,657 INFO    [main] [ConsoleStatus] Starting [47/85]: InstanceManager 
2017-02-07 21:40:12,657 INFO    [main] [ConsoleStatus] Starting [48/85]: MachineDriverLoader 
2017-02-07 21:40:12,659 INFO    [main] [ConsoleStatus] Starting [49/85]: MachineLauncher 
2017-02-07 21:40:12,659 INFO    [main] [ConsoleStatus] Starting [50/85]: MountRemove 
2017-02-07 21:40:12,660 INFO    [main] [ConsoleStatus] Starting [51/85]: PostInstancePurge 
2017-02-07 21:40:12,660 INFO    [main] [ConsoleStatus] Starting [52/85]: PostStartLabelsProviders 
2017-02-07 21:40:12,660 INFO    [main] [ConsoleStatus] Starting [53/85]: ProjectMemberResourceManager 
2017-02-07 21:40:12,660 INFO    [main] [ConsoleStatus] Starting [54/85]: ProjectResourceManager 
2017-02-07 21:40:12,660 INFO    [main] [ConsoleStatus] Starting [55/85]: PullTaskCreate 
2017-02-07 21:40:12,660 INFO    [main] [ConsoleStatus] Starting [56/85]: RestoreFromBackup 
2017-02-07 21:40:12,661 INFO    [main] [ConsoleStatus] Starting [57/85]: RevertToSnapshot 
2017-02-07 21:40:12,661 INFO    [main] [ConsoleStatus] Starting [58/85]: SampleDataStartupV10 
2017-02-07 21:40:12,667 INFO    [main] [ConsoleStatus] Starting [59/85]: SampleDataStartupV11 
2017-02-07 21:40:12,670 INFO    [main] [ConsoleStatus] Starting [60/85]: SampleDataStartupV12 
2017-02-07 21:40:12,673 INFO    [main] [ConsoleStatus] Starting [61/85]: SampleDataStartupV13 
2017-02-07 21:40:12,676 INFO    [main] [ConsoleStatus] Starting [62/85]: SampleDataStartupV5 
2017-02-07 21:40:12,680 INFO    [main] [ConsoleStatus] Starting [63/85]: SampleDataStartupV6 
2017-02-07 21:40:12,683 INFO    [main] [ConsoleStatus] Starting [64/85]: SampleDataStartupV7 
2017-02-07 21:40:12,687 INFO    [main] [ConsoleStatus] Starting [65/85]: SampleDataStartupV8 
2017-02-07 21:40:12,689 INFO    [main] [ConsoleStatus] Starting [66/85]: SampleDataStartupV9 
2017-02-07 21:40:12,692 INFO    [main] [ConsoleStatus] Starting [67/85]: SecretManager 
2017-02-07 21:40:12,692 INFO    [main] [ConsoleStatus] Starting [68/85]: SecretsApiLauncher 
2017-02-07 21:40:12,692 INFO    [main] [ConsoleStatus] Starting [69/85]: ServiceManager 
2017-02-07 21:40:12,692 INFO    [main] [ConsoleStatus] Starting [70/85]: SettingManager 
2017-02-07 21:40:12,692 INFO    [main] [ConsoleStatus] Starting [71/85]: SnapshotCreate 
2017-02-07 21:40:12,692 INFO    [main] [ConsoleStatus] Starting [72/85]: SnapshotRemove 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [73/85]: StackAgentHandler 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [74/85]: StackAgentHandler 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [75/85]: StackAgentHandler 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [76/85]: StackAgentHandler 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [77/85]: StackAgentHandler 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [78/85]: TelemetryLauncher 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [79/85]: VolumeManager 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [80/85]: VolumeRemove 
2017-02-07 21:40:12,693 INFO    [main] [ConsoleStatus] Starting [81/85]: VolumeStoragePoolMapActivate 
2017-02-07 21:40:12,694 INFO    [main] [ConsoleStatus] Starting [82/85]: VolumeStoragePoolMapRemove 
2017-02-07 21:40:12,694 INFO    [main] [ConsoleStatus] Starting [83/85]: WebhookServiceLauncher 
2017-02-07 21:40:12,694 INFO    [main] [ConsoleStatus] Starting [84/85]: WebsocketProxyLauncher 
2017-02-07 21:40:12,694 INFO    [main] [ConsoleStatus] Starting [85/85]: project.template.reload 
2017-02-07 21:40:12,844 INFO    [main] [ConsoleStatus] [DONE ] [49153ms] Startup Succeeded, Listening on port 8081 
time="2017-02-07T21:40:12Z" level=info msg="Request to refresh catalog" 
time="2017-02-07T21:40:12Z" level=info msg="Using catalog library=https://git.rancher.io/rancher-catalog.git" 
time="2017-02-07T21:40:12Z" level=info msg="Using catalog community=https://git.rancher.io/community-catalog.git" 
time="2017-02-07T21:40:13Z" level=info msg="Starting rancher-compose-executor" version=v0.12.2 
time="2017-02-07T21:40:13Z" level=fatal msg="Unable to create event router" error="Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused" 
time="2017-02-07T21:40:13Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v1: dial tcp [::1]:8080: getsockopt: connection refused" 
time="2017-02-07T21:40:15Z" level=error msg="Failed to pull the catalog from git repo https://git.rancher.io/community-catalog.git, error: exit status 1" 
time="2017-02-07T21:40:15Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt" 
time="2017-02-07T21:40:15Z" level=info msg="Webhook service listening on 8085" 
time="2017-02-07T21:40:15Z" level=info msg="Setting log level" logLevel=info 
time="2017-02-07T21:40:15Z" level=info msg="Starting go-machine-service..." gitcommit=v0.35.0 
time="2017-02-07T21:40:15Z" level=info msg="Waiting for handler registration (1/2)" 
time="2017-02-07T21:40:15Z" level=fatal msg="Exiting go-machine-service: Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused" 
time="2017-02-07T21:40:15Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [7]." 
time="2017-02-07T21:40:15Z" level=info msg="Master config file: master.conf" 
time="2017-02-07T21:40:15Z" level=info msg="Starting rancher-compose-executor" version=v0.12.2 
time="2017-02-07T21:40:16Z" level=info msg="Downloading certificate from http://localhost:8081/v1/credentials/1c1/certificate" 
time="2017-02-07T21:40:16Z" level=info msg="Setting log level" logLevel=info 
time="2017-02-07T21:40:16Z" level=info msg="Starting go-machine-service..." gitcommit=v0.35.0 
time="2017-02-07T21:40:16Z" level=info msg="Waiting for handler registration (1/2)" 
time="2017-02-07T21:40:18Z" level=info msg="Starting Rancher Auth service" 
time="2017-02-07T21:40:19Z" level=info msg=" Auth provider configured githubconfig" 
time="2017-02-07T21:40:19Z" level=info msg="Listening on :8090" 
time="2017-02-07T21:40:19Z" level=info msg="Initializing event router" workerCount=10 
time="2017-02-07T21:40:20Z" level=info msg="Initializing event router" workerCount=10 
time="2017-02-07T21:40:20Z" level=info msg="Connection established" 
time="2017-02-07T21:40:20Z" level=info msg="Starting websocket pings" 
time="2017-02-07T21:40:20Z" level=info msg="Connection established" 
time="2017-02-07T21:40:20Z" level=info msg="Starting websocket pings" 
time="2017-02-07T21:40:20Z" level=info msg="Waiting for handler registration (2/2)" 
time="2017-02-07T21:40:20Z" level=info msg="Initializing event router" workerCount=10 
time="2017-02-07T21:40:20Z" level=info msg="Connection established" 
time="2017-02-07T21:40:20Z" level=info msg="Starting websocket pings" 
time="2017-02-07T21:40:20Z" level=info msg="Installing builtin drivers" 
time="2017-02-07T21:40:20Z" level=info msg="Downloading all drivers" 
time="2017-02-07T21:40:21Z" level=info msg="Download https://github.com/packethost/docker-machine-driver-packet/releases/download/v0.1.2/docker-machine-driver-packet_linux-amd64.zip" 
time="2017-02-07T21:40:24Z" level=info msg="Found driver docker-machine-driver-packet" 
time="2017-02-07T21:40:24Z" level=info msg="Copying /var/lib/cattle/machine-drivers/1f7058341420e2f525168052818c3f819ff78e9ca5f57d5a650a049bcd5945e9-docker-machine-driver-packet => /usr/local/bin/docker-machine-driver-packet" 
time="2017-02-07T21:40:24Z" level=info msg="Done downloading all drivers" 
# output of docker inspect rancher-server
[
    {
        "Id": "e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d",
        "Created": "2017-02-07T21:39:21.913836095Z",
        "Path": "/usr/bin/entry",
        "Args": [
            "/usr/bin/s6-svscan",
            "/service"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 31610,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2017-02-07T21:39:23.032979732Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:dcfac2209d79a8392543a6df9faf7207fe9c96f8659576776e3eb386bd873528",
        "ResolvConfPath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/hostname",
        "HostsPath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/hosts",
        "LogPath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d-json.log",
        "Name": "/rancher-server",
        "RestartCount": 0,
        "Driver": "devicemapper",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "devicemapper",
            "Data": {
                "DeviceId": "72",
                "DeviceName": "docker-202:1-394679-317792deb7bc1afc7d95491baf0305b149027f82f488707a277ac0bc86c1031d",
                "DeviceSize": "10737418240"
            }
        },
        "Mounts": [
            {
                "Name": "95fd5eee4122512cd4da414397d40651ead5ed1c1e1a3c6c705e5e064b513904",
                "Source": "/var/lib/docker/volumes/95fd5eee4122512cd4da414397d40651ead5ed1c1e1a3c6c705e5e064b513904/_data",
                "Destination": "/var/log/mysql",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Name": "a84d47c9ffe034522dde80f11309cf5312b6041e5223031eefb8a20ca446b80e",
                "Source": "/var/lib/docker/volumes/a84d47c9ffe034522dde80f11309cf5312b6041e5223031eefb8a20ca446b80e/_data",
                "Destination": "/var/lib/cattle",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Name": "95249d330778f4381ea21f9858de6622bddfd8045326f078fde0d651ebf07755",
                "Source": "/var/lib/docker/volumes/95249d330778f4381ea21f9858de6622bddfd8045326f078fde0d651ebf07755/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "e6ebd5cb70de",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "8080/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "CATTLE_DB_CATTLE_MYSQL_HOST=<my_db_host>",
                "CATTLE_DB_CATTLE_MYSQL_PORT=3306",
                "CATTLE_DB_CATTLE_MYSQL_NAME=<my_db_name>",
                "CATTLE_DB_CATTLE_USERNAME=<my_db_user>",
                "CATTLE_DB_CATTLE_PASSWORD=<my_db_password>",
                "VIRTUAL_HOST=<my_hostname>",
                "VIRTUAL_PORT=8080",
                "LETSENCRYPT_HOST=<my_hostname>",
                "LETSENCRYPT_EMAIL=<my_email>",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=en_US.UTF-8",
                "LANGUAGE=en_US:en",
                "LC_ALL=en_US.UTF-8",
                "DEBIAN_FRONTEND=noninteractive",
                "CATTLE_HOME=/var/lib/cattle",
                "DEFAULT_CATTLE_API_UI_INDEX=//releases.rancher.com/ui/1.4.6",
                "CATTLE_API_UI_URL=//releases.rancher.com/api-ui/1.0.8",
                "CATTLE_DB_CATTLE_DATABASE=mysql",
                "CATTLE_USE_LOCAL_ARTIFACTS=true",
                "no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com",
                "S6_SERVICE_DIR=/service",
                "CATTLE_HOST_API_PROXY_MODE=embedded",
                "CATTLE_RANCHER_SERVER_VERSION=v1.4.0",
                "CATTLE_RANCHER_COMPOSE_VERSION=v0.12.2",
                "DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-linux-amd64-v0.12.2.tar.gz",
                "DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-darwin-amd64-v0.12.2.tar.gz",
                "DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-windows-386-v0.12.2.zip",
                "CATTLE_RANCHER_CLI_VERSION=v0.4.1",
                "DEFAULT_CATTLE_RANCHER_CLI_LINUX_URL=https://releases.rancher.com/cli/v0.4.1/rancher-linux-amd64-v0.4.1.tar.gz",
                "DEFAULT_CATTLE_RANCHER_CLI_DARWIN_URL=https://releases.rancher.com/cli/v0.4.1/rancher-darwin-amd64-v0.4.1.tar.gz",
                "DEFAULT_CATTLE_RANCHER_CLI_WINDOWS_URL=https://releases.rancher.com/cli/v0.4.1/rancher-windows-386-v0.4.1.zip",
                "DEFAULT_CATTLE_CATALOG_URL=community=https://git.rancher.io/community-catalog.git,library=https://git.rancher.io/rancher-catalog.git",
                "CATTLE_CATTLE_VERSION=v0.176.8",
                "DEFAULT_CATTLE_API_UI_JS_URL=/api-ui/ui.min.js",
                "DEFAULT_CATTLE_API_UI_CSS_URL=/api-ui/ui.min.css",
                "DEFAULT_CATTLE_MACHINE_EXECUTE=true",
                "DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE=true",
                "DEFAULT_CATTLE_CATALOG_EXECUTE=true",
                "DEFAULT_CATTLE_AUTH_SERVICE_EXECUTE=true",
                "DEFAULT_CATTLE_WEBHOOK_SERVICE_EXECUTE=true",
                "DEFAULT_CATTLE_SECRETS_API_EXECUTE=true",
                "CATTLE_RANCHER_SERVER_IMAGE=rancher/server"
            ],
            "Cmd": [
                "/usr/bin/s6-svscan",
                "/service"
            ],
            "Image": "rancher/server:v1.4.0",
            "Volumes": {
                "/var/lib/cattle": {},
                "/var/lib/mysql": {},
                "/var/log/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/usr/bin/entry"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "75040e3b9459631172bcda137560c63a0bcada4ed8bc1187ac59283e0b50f856",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3306/tcp": null,
                "8080/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/75040e3b9459",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "db44e4b02038862fdaf621f42729be938174de14cecc1c605ae7f1a5b74f23d1",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "8e97fac0070ecc7418a2dcc2b08ec5bf7a854616b38285e328b6db0bba241022",
                    "EndpointID": "db44e4b02038862fdaf621f42729be938174de14cecc1c605ae7f1a5b74f23d1",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02"
                }
            }
        }
    }
]

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 25 (2 by maintainers)

Most upvoted comments

@Niceplace @jsalva if you see these errors solely on boot and then they go away, they can be ignored. Here’s what’s happening: In addition to the primary java API server, rancher/server starts up a number of services/processes. One of those is a reverse proxy that sits in front of the java API server (which is actually listening on port 8081) and listens on port 8080. It also starts microservices that make requests to localhost:8080. But, it doesn’t intelligently wait for the reverse proxy to be up before starting the microservices. It instead relies on the fact that those microservices will keep trying to connect until they do so successfully.

Due to the obvious uptick in people reporting this issue, something subtle might have changed recently to cause the reverse proxy to not start as quickly. To alleviate, we might have to add some logic to mitigate this scenario.

From what I can tell in this issue and #7523, most people have a functioning Rancher and are just concerned with the error message.

If that is NOT the case for you and you have an actual functionality problem, could you open a separate issue describing your issue?