rancher: Rancher Server 1.4.0 unable to make calls to api (getsockopt: connection refused)
Rancher Versions: Server: v1.4.0
Docker Version: Docker version 1.12.6, build 7392c3b/1.12.6
OS and where are the hosts located? (cloud, bare metal, etc): Amazon Linux (AWS)
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB) Single node, external DB (RDS)
Environment Type: (Cattle/Kubernetes/Swarm/Mesos) Cattle
Steps to Reproduce:
docker run -d --restart=always --name=rancher-server --expose 8080/tcp \
-e CATTLE_DB_CATTLE_MYSQL_HOST=<my_db_host> \
-e CATTLE_DB_CATTLE_MYSQL_PORT=3306 \
-e CATTLE_DB_CATTLE_MYSQL_NAME=<my_db_name> \
-e CATTLE_DB_CATTLE_USERNAME=<my_db_user> \
-e CATTLE_DB_CATTLE_PASSWORD=<my_db_password> \
-e VIRTUAL_HOST=<my_hostname> \
-e VIRTUAL_PORT=8080 \
rancher/server:v1.4.0
Results:
every request to localhost:8080 fails with something along the lines of Unable to create event router" error="Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused
# output of docker logs rancher-server
# requests to localhost:8080 fail (see below)
CATTLE_AGENT_PACKAGE_HOST_API_URL=/usr/share/cattle/artifacts/host-api.tar.gz
CATTLE_AGENT_PACKAGE_PYTHON_AGENT_URL=/usr/share/cattle/artifacts/go-agent.tar.gz
CATTLE_API_UI_URL=//releases.rancher.com/api-ui/1.0.8
CATTLE_CATTLE_VERSION=v0.176.8
CATTLE_DB_CATTLE_DATABASE=mysql
CATTLE_DB_CATTLE_MYSQL_HOST=<my_db_host>
CATTLE_DB_CATTLE_MYSQL_NAME=<my_db_name>
CATTLE_DB_CATTLE_MYSQL_PORT=3306
CATTLE_DB_CATTLE_USERNAME=<my_db_user>
CATTLE_GRAPHITE_HOST=
CATTLE_GRAPHITE_PORT=
CATTLE_HOME=/var/lib/cattle
CATTLE_HOST_API_PROXY_MODE=embedded
CATTLE_LOGBACK_OUTPUT_GELF_HOST=
CATTLE_LOGBACK_OUTPUT_GELF_PORT=
CATTLE_RANCHER_CLI_VERSION=v0.4.1
CATTLE_RANCHER_COMPOSE_VERSION=v0.12.2
CATTLE_RANCHER_SERVER_IMAGE=rancher/server
CATTLE_RANCHER_SERVER_VERSION=v1.4.0
CATTLE_USE_LOCAL_ARTIFACTS=true
DEFAULT_CATTLE_API_UI_CSS_URL=/api-ui/ui.min.css
DEFAULT_CATTLE_API_UI_INDEX=//releases.rancher.com/ui/1.4.6
DEFAULT_CATTLE_API_UI_JS_URL=/api-ui/ui.min.js
DEFAULT_CATTLE_AUTH_SERVICE_EXECUTE=true
DEFAULT_CATTLE_CATALOG_EXECUTE=true
DEFAULT_CATTLE_CATALOG_URL=community=https://git.rancher.io/community-catalog.git,library=https://git.rancher.io/rancher-catalog.git
DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE=true
DEFAULT_CATTLE_MACHINE_EXECUTE=true
DEFAULT_CATTLE_RANCHER_CLI_DARWIN_URL=https://releases.rancher.com/cli/v0.4.1/rancher-darwin-amd64-v0.4.1.tar.gz
DEFAULT_CATTLE_RANCHER_CLI_LINUX_URL=https://releases.rancher.com/cli/v0.4.1/rancher-linux-amd64-v0.4.1.tar.gz
DEFAULT_CATTLE_RANCHER_CLI_WINDOWS_URL=https://releases.rancher.com/cli/v0.4.1/rancher-windows-386-v0.4.1.zip
DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-darwin-amd64-v0.12.2.tar.gz
DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-linux-amd64-v0.12.2.tar.gz
DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-windows-386-v0.12.2.zip
DEFAULT_CATTLE_SECRETS_API_EXECUTE=true
DEFAULT_CATTLE_WEBHOOK_SERVICE_EXECUTE=true
21:39:29.194 [main] INFO ConsoleStatus - Loading configuration
2017-02-07 21:39:37,309 INFO [main] [ConsoleStatus] Starting DB migration
2017-02-07 21:39:40,805 INFO [main] [ConsoleStatus] DB migration done
2017-02-07 21:39:41,492 INFO [main] [ConsoleStatus] Cluster membership changed [127.0.0.1:9345]
2017-02-07 21:39:41,495 INFO [main] [ConsoleStatus] Checking cluster state on start-up
2017-02-07 21:39:41,495 INFO [main] [ConsoleStatus] Waiting to become master
2017-02-07 21:40:05,058 INFO [main] [ConsoleStatus] Loading processes
2017-02-07 21:40:05,772 INFO [main] [ConsoleStatus] Starting [1/85]: LockDelegatorImpl
2017-02-07 21:40:05,777 INFO [main] [ConsoleStatus] Starting [2/85]: AnnotatedListenerRegistration
2017-02-07 21:40:05,797 INFO [main] [ConsoleStatus] Starting [3/85]: EventService
2017-02-07 21:40:05,797 INFO [main] [ConsoleStatus] Starting [4/85]: DefaultObjectMetaDataManager
2017-02-07 21:40:07,486 INFO [main] [ConsoleStatus] Starting [5/85]: JsonDefaultsProvider
2017-02-07 21:40:07,590 INFO [main] [ConsoleStatus] Starting [6/85]: ObjectDefaultsPostInstantiationHandler
2017-02-07 21:40:07,590 INFO [main] [ConsoleStatus] Starting [7/85]: DefaultProcessManager
2017-02-07 21:40:07,593 INFO [main] [ConsoleStatus] Starting [8/85]: SampleDataStartupV3
2017-02-07 21:40:07,607 INFO [main] [ConsoleStatus] Starting [9/85]: TaskManagerImpl
2017-02-07 21:40:07,787 INFO [main] [ConsoleStatus] Starting [10/85]: ServiceAccountCreateStartup
2017-02-07 21:40:07,818 INFO [main] [ConsoleStatus] Starting [11/85]: SchemaFactory:v1-admin
2017-02-07 21:40:08,099 INFO [main] [ConsoleStatus] Starting [12/85]: SchemaFactory:v1-agent
2017-02-07 21:40:08,105 INFO [main] [ConsoleStatus] Starting [13/85]: SchemaFactory:v1-agentRegister
2017-02-07 21:40:08,112 INFO [main] [ConsoleStatus] Starting [14/85]: SchemaFactory:v1-base
2017-02-07 21:40:08,381 INFO [main] [ConsoleStatus] Starting [15/85]: SchemaFactory:v1-member
2017-02-07 21:40:08,517 INFO [main] [ConsoleStatus] Starting [16/85]: SchemaFactory:v1-owner
2017-02-07 21:40:08,678 INFO [main] [ConsoleStatus] Starting [17/85]: SchemaFactory:v1-project
2017-02-07 21:40:08,802 INFO [main] [ConsoleStatus] Starting [18/85]: SchemaFactory:v1-projectadmin
2017-02-07 21:40:08,920 INFO [main] [ConsoleStatus] Starting [19/85]: SchemaFactory:v1-readAdmin
2017-02-07 21:40:09,044 INFO [main] [ConsoleStatus] Starting [20/85]: SchemaFactory:v1-readonly
2017-02-07 21:40:09,108 INFO [main] [ConsoleStatus] Starting [21/85]: SchemaFactory:v1-register
2017-02-07 21:40:09,116 INFO [main] [ConsoleStatus] Starting [22/85]: SchemaFactory:v1-restricted
2017-02-07 21:40:09,218 INFO [main] [ConsoleStatus] Starting [23/85]: SchemaFactory:v1-service
2017-02-07 21:40:09,285 INFO [main] [ConsoleStatus] Starting [24/85]: SchemaFactory:v1-token
2017-02-07 21:40:09,289 INFO [main] [ConsoleStatus] Starting [25/85]: SchemaFactory:v1-user
2017-02-07 21:40:09,354 INFO [main] [ConsoleStatus] Starting [26/85]: AgentBasedProcessHandler
2017-02-07 21:40:09,354 INFO [main] [ConsoleStatus] Starting [27/85]: AgentHostStateUpdate
2017-02-07 21:40:09,355 INFO [main] [ConsoleStatus] Starting [28/85]: AuthServiceLauncher
2017-02-07 21:40:09,365 INFO [main] [ConsoleStatus] Starting [29/85]: BackupCreate
2017-02-07 21:40:09,367 INFO [main] [ConsoleStatus] Starting [30/85]: BackupRemove
2017-02-07 21:40:09,367 INFO [main] [ConsoleStatus] Starting [31/85]: CatalogLauncher
2017-02-07 21:40:09,367 INFO [main] [ConsoleStatus] Starting [32/85]: ComposeExecutorLauncher
2017-02-07 21:40:09,372 INFO [main] [ConsoleStatus] Starting [33/85]: ConfigItemRegistryImpl
2017-02-07 21:40:09,562 INFO [main] [ConsoleStatus] Starting [34/85]: ConfigItemServerImpl
2017-02-07 21:40:09,884 INFO [main] [ConsoleStatus] Starting [35/85]: ConfigUpdatePublisher
2017-02-07 21:40:09,884 INFO [main] [ConsoleStatus] Starting [36/85]: DataManager
2017-02-07 21:40:09,884 INFO [main] [ConsoleStatus] Starting [37/85]: DefaultAuthorizationProvider
time="2017-02-07T21:40:11Z" level=info msg="Starting rancher-compose-executor" version=v0.12.2
time="2017-02-07T21:40:11Z" level=fatal msg="Unable to create event router" error="Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused"
time="2017-02-07T21:40:11Z" level=info msg="Starting Rancher Catalog service"
time="2017-02-07T21:40:11Z" level=info msg="Using catalog community=https://git.rancher.io/community-catalog.git"
time="2017-02-07T21:40:11Z" level=info msg="Using catalog library=https://git.rancher.io/rancher-catalog.git"
time="2017-02-07T21:40:11Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v1: dial tcp [::1]:8080: getsockopt: connection refused"
2017-02-07 21:40:12,647 INFO [main] [ConsoleStatus] Starting [38/85]: DefaultJooqResourceManager
2017-02-07 21:40:12,648 INFO [main] [ConsoleStatus] Starting [39/85]: DefaultObjectSerializerFactoryImpl
2017-02-07 21:40:12,648 INFO [main] [ConsoleStatus] Starting [40/85]: DockerComposeStackHandler
2017-02-07 21:40:12,649 INFO [main] [ConsoleStatus] Starting [41/85]: DynamicExtensionManager
2017-02-07 21:40:12,649 INFO [main] [ConsoleStatus] Starting [42/85]: ExtensionResourceManagerLocator
2017-02-07 21:40:12,649 INFO [main] [ConsoleStatus] Starting [43/85]: HostApiRSAKeyProvider
2017-02-07 21:40:12,652 INFO [main] [ConsoleStatus] Starting [44/85]: ImageStoragePoolMapActivate
2017-02-07 21:40:12,652 INFO [main] [ConsoleStatus] Starting [45/85]: InstanceHostMapActivate
2017-02-07 21:40:12,653 INFO [main] [ConsoleStatus] Starting [46/85]: InstanceHostMapDeactivate
2017-02-07 21:40:12,657 INFO [main] [ConsoleStatus] Starting [47/85]: InstanceManager
2017-02-07 21:40:12,657 INFO [main] [ConsoleStatus] Starting [48/85]: MachineDriverLoader
2017-02-07 21:40:12,659 INFO [main] [ConsoleStatus] Starting [49/85]: MachineLauncher
2017-02-07 21:40:12,659 INFO [main] [ConsoleStatus] Starting [50/85]: MountRemove
2017-02-07 21:40:12,660 INFO [main] [ConsoleStatus] Starting [51/85]: PostInstancePurge
2017-02-07 21:40:12,660 INFO [main] [ConsoleStatus] Starting [52/85]: PostStartLabelsProviders
2017-02-07 21:40:12,660 INFO [main] [ConsoleStatus] Starting [53/85]: ProjectMemberResourceManager
2017-02-07 21:40:12,660 INFO [main] [ConsoleStatus] Starting [54/85]: ProjectResourceManager
2017-02-07 21:40:12,660 INFO [main] [ConsoleStatus] Starting [55/85]: PullTaskCreate
2017-02-07 21:40:12,660 INFO [main] [ConsoleStatus] Starting [56/85]: RestoreFromBackup
2017-02-07 21:40:12,661 INFO [main] [ConsoleStatus] Starting [57/85]: RevertToSnapshot
2017-02-07 21:40:12,661 INFO [main] [ConsoleStatus] Starting [58/85]: SampleDataStartupV10
2017-02-07 21:40:12,667 INFO [main] [ConsoleStatus] Starting [59/85]: SampleDataStartupV11
2017-02-07 21:40:12,670 INFO [main] [ConsoleStatus] Starting [60/85]: SampleDataStartupV12
2017-02-07 21:40:12,673 INFO [main] [ConsoleStatus] Starting [61/85]: SampleDataStartupV13
2017-02-07 21:40:12,676 INFO [main] [ConsoleStatus] Starting [62/85]: SampleDataStartupV5
2017-02-07 21:40:12,680 INFO [main] [ConsoleStatus] Starting [63/85]: SampleDataStartupV6
2017-02-07 21:40:12,683 INFO [main] [ConsoleStatus] Starting [64/85]: SampleDataStartupV7
2017-02-07 21:40:12,687 INFO [main] [ConsoleStatus] Starting [65/85]: SampleDataStartupV8
2017-02-07 21:40:12,689 INFO [main] [ConsoleStatus] Starting [66/85]: SampleDataStartupV9
2017-02-07 21:40:12,692 INFO [main] [ConsoleStatus] Starting [67/85]: SecretManager
2017-02-07 21:40:12,692 INFO [main] [ConsoleStatus] Starting [68/85]: SecretsApiLauncher
2017-02-07 21:40:12,692 INFO [main] [ConsoleStatus] Starting [69/85]: ServiceManager
2017-02-07 21:40:12,692 INFO [main] [ConsoleStatus] Starting [70/85]: SettingManager
2017-02-07 21:40:12,692 INFO [main] [ConsoleStatus] Starting [71/85]: SnapshotCreate
2017-02-07 21:40:12,692 INFO [main] [ConsoleStatus] Starting [72/85]: SnapshotRemove
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [73/85]: StackAgentHandler
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [74/85]: StackAgentHandler
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [75/85]: StackAgentHandler
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [76/85]: StackAgentHandler
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [77/85]: StackAgentHandler
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [78/85]: TelemetryLauncher
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [79/85]: VolumeManager
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [80/85]: VolumeRemove
2017-02-07 21:40:12,693 INFO [main] [ConsoleStatus] Starting [81/85]: VolumeStoragePoolMapActivate
2017-02-07 21:40:12,694 INFO [main] [ConsoleStatus] Starting [82/85]: VolumeStoragePoolMapRemove
2017-02-07 21:40:12,694 INFO [main] [ConsoleStatus] Starting [83/85]: WebhookServiceLauncher
2017-02-07 21:40:12,694 INFO [main] [ConsoleStatus] Starting [84/85]: WebsocketProxyLauncher
2017-02-07 21:40:12,694 INFO [main] [ConsoleStatus] Starting [85/85]: project.template.reload
2017-02-07 21:40:12,844 INFO [main] [ConsoleStatus] [DONE ] [49153ms] Startup Succeeded, Listening on port 8081
time="2017-02-07T21:40:12Z" level=info msg="Request to refresh catalog"
time="2017-02-07T21:40:12Z" level=info msg="Using catalog library=https://git.rancher.io/rancher-catalog.git"
time="2017-02-07T21:40:12Z" level=info msg="Using catalog community=https://git.rancher.io/community-catalog.git"
time="2017-02-07T21:40:13Z" level=info msg="Starting rancher-compose-executor" version=v0.12.2
time="2017-02-07T21:40:13Z" level=fatal msg="Unable to create event router" error="Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused"
time="2017-02-07T21:40:13Z" level=fatal msg="Failed to configure cattle client: Get http://localhost:8080/v1: dial tcp [::1]:8080: getsockopt: connection refused"
time="2017-02-07T21:40:15Z" level=error msg="Failed to pull the catalog from git repo https://git.rancher.io/community-catalog.git, error: exit status 1"
time="2017-02-07T21:40:15Z" level=info msg="Downloading key from http://localhost:8081/v1/scripts/api.crt"
time="2017-02-07T21:40:15Z" level=info msg="Webhook service listening on 8085"
time="2017-02-07T21:40:15Z" level=info msg="Setting log level" logLevel=info
time="2017-02-07T21:40:15Z" level=info msg="Starting go-machine-service..." gitcommit=v0.35.0
time="2017-02-07T21:40:15Z" level=info msg="Waiting for handler registration (1/2)"
time="2017-02-07T21:40:15Z" level=fatal msg="Exiting go-machine-service: Get http://localhost:8080/v2-beta: dial tcp [::1]:8080: getsockopt: connection refused"
time="2017-02-07T21:40:15Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [7]."
time="2017-02-07T21:40:15Z" level=info msg="Master config file: master.conf"
time="2017-02-07T21:40:15Z" level=info msg="Starting rancher-compose-executor" version=v0.12.2
time="2017-02-07T21:40:16Z" level=info msg="Downloading certificate from http://localhost:8081/v1/credentials/1c1/certificate"
time="2017-02-07T21:40:16Z" level=info msg="Setting log level" logLevel=info
time="2017-02-07T21:40:16Z" level=info msg="Starting go-machine-service..." gitcommit=v0.35.0
time="2017-02-07T21:40:16Z" level=info msg="Waiting for handler registration (1/2)"
time="2017-02-07T21:40:18Z" level=info msg="Starting Rancher Auth service"
time="2017-02-07T21:40:19Z" level=info msg=" Auth provider configured githubconfig"
time="2017-02-07T21:40:19Z" level=info msg="Listening on :8090"
time="2017-02-07T21:40:19Z" level=info msg="Initializing event router" workerCount=10
time="2017-02-07T21:40:20Z" level=info msg="Initializing event router" workerCount=10
time="2017-02-07T21:40:20Z" level=info msg="Connection established"
time="2017-02-07T21:40:20Z" level=info msg="Starting websocket pings"
time="2017-02-07T21:40:20Z" level=info msg="Connection established"
time="2017-02-07T21:40:20Z" level=info msg="Starting websocket pings"
time="2017-02-07T21:40:20Z" level=info msg="Waiting for handler registration (2/2)"
time="2017-02-07T21:40:20Z" level=info msg="Initializing event router" workerCount=10
time="2017-02-07T21:40:20Z" level=info msg="Connection established"
time="2017-02-07T21:40:20Z" level=info msg="Starting websocket pings"
time="2017-02-07T21:40:20Z" level=info msg="Installing builtin drivers"
time="2017-02-07T21:40:20Z" level=info msg="Downloading all drivers"
time="2017-02-07T21:40:21Z" level=info msg="Download https://github.com/packethost/docker-machine-driver-packet/releases/download/v0.1.2/docker-machine-driver-packet_linux-amd64.zip"
time="2017-02-07T21:40:24Z" level=info msg="Found driver docker-machine-driver-packet"
time="2017-02-07T21:40:24Z" level=info msg="Copying /var/lib/cattle/machine-drivers/1f7058341420e2f525168052818c3f819ff78e9ca5f57d5a650a049bcd5945e9-docker-machine-driver-packet => /usr/local/bin/docker-machine-driver-packet"
time="2017-02-07T21:40:24Z" level=info msg="Done downloading all drivers"
# output of docker inspect rancher-server
[
{
"Id": "e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d",
"Created": "2017-02-07T21:39:21.913836095Z",
"Path": "/usr/bin/entry",
"Args": [
"/usr/bin/s6-svscan",
"/service"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 31610,
"ExitCode": 0,
"Error": "",
"StartedAt": "2017-02-07T21:39:23.032979732Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:dcfac2209d79a8392543a6df9faf7207fe9c96f8659576776e3eb386bd873528",
"ResolvConfPath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/hostname",
"HostsPath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/hosts",
"LogPath": "/var/lib/docker/containers/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d/e6ebd5cb70de3733b234c47db7e326379b84100672fa5a6500b6033c9963e70d-json.log",
"Name": "/rancher-server",
"RestartCount": 0,
"Driver": "devicemapper",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Name": "devicemapper",
"Data": {
"DeviceId": "72",
"DeviceName": "docker-202:1-394679-317792deb7bc1afc7d95491baf0305b149027f82f488707a277ac0bc86c1031d",
"DeviceSize": "10737418240"
}
},
"Mounts": [
{
"Name": "95fd5eee4122512cd4da414397d40651ead5ed1c1e1a3c6c705e5e064b513904",
"Source": "/var/lib/docker/volumes/95fd5eee4122512cd4da414397d40651ead5ed1c1e1a3c6c705e5e064b513904/_data",
"Destination": "/var/log/mysql",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "a84d47c9ffe034522dde80f11309cf5312b6041e5223031eefb8a20ca446b80e",
"Source": "/var/lib/docker/volumes/a84d47c9ffe034522dde80f11309cf5312b6041e5223031eefb8a20ca446b80e/_data",
"Destination": "/var/lib/cattle",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Name": "95249d330778f4381ea21f9858de6622bddfd8045326f078fde0d651ebf07755",
"Source": "/var/lib/docker/volumes/95249d330778f4381ea21f9858de6622bddfd8045326f078fde0d651ebf07755/_data",
"Destination": "/var/lib/mysql",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "e6ebd5cb70de",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"3306/tcp": {},
"8080/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"CATTLE_DB_CATTLE_MYSQL_HOST=<my_db_host>",
"CATTLE_DB_CATTLE_MYSQL_PORT=3306",
"CATTLE_DB_CATTLE_MYSQL_NAME=<my_db_name>",
"CATTLE_DB_CATTLE_USERNAME=<my_db_user>",
"CATTLE_DB_CATTLE_PASSWORD=<my_db_password>",
"VIRTUAL_HOST=<my_hostname>",
"VIRTUAL_PORT=8080",
"LETSENCRYPT_HOST=<my_hostname>",
"LETSENCRYPT_EMAIL=<my_email>",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=en_US.UTF-8",
"LANGUAGE=en_US:en",
"LC_ALL=en_US.UTF-8",
"DEBIAN_FRONTEND=noninteractive",
"CATTLE_HOME=/var/lib/cattle",
"DEFAULT_CATTLE_API_UI_INDEX=//releases.rancher.com/ui/1.4.6",
"CATTLE_API_UI_URL=//releases.rancher.com/api-ui/1.0.8",
"CATTLE_DB_CATTLE_DATABASE=mysql",
"CATTLE_USE_LOCAL_ARTIFACTS=true",
"no_proxy=localhost,127.0.0.1,localaddress,.localdomain.com",
"S6_SERVICE_DIR=/service",
"CATTLE_HOST_API_PROXY_MODE=embedded",
"CATTLE_RANCHER_SERVER_VERSION=v1.4.0",
"CATTLE_RANCHER_COMPOSE_VERSION=v0.12.2",
"DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-linux-amd64-v0.12.2.tar.gz",
"DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-darwin-amd64-v0.12.2.tar.gz",
"DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL=https://releases.rancher.com/compose/v0.12.2/rancher-compose-windows-386-v0.12.2.zip",
"CATTLE_RANCHER_CLI_VERSION=v0.4.1",
"DEFAULT_CATTLE_RANCHER_CLI_LINUX_URL=https://releases.rancher.com/cli/v0.4.1/rancher-linux-amd64-v0.4.1.tar.gz",
"DEFAULT_CATTLE_RANCHER_CLI_DARWIN_URL=https://releases.rancher.com/cli/v0.4.1/rancher-darwin-amd64-v0.4.1.tar.gz",
"DEFAULT_CATTLE_RANCHER_CLI_WINDOWS_URL=https://releases.rancher.com/cli/v0.4.1/rancher-windows-386-v0.4.1.zip",
"DEFAULT_CATTLE_CATALOG_URL=community=https://git.rancher.io/community-catalog.git,library=https://git.rancher.io/rancher-catalog.git",
"CATTLE_CATTLE_VERSION=v0.176.8",
"DEFAULT_CATTLE_API_UI_JS_URL=/api-ui/ui.min.js",
"DEFAULT_CATTLE_API_UI_CSS_URL=/api-ui/ui.min.css",
"DEFAULT_CATTLE_MACHINE_EXECUTE=true",
"DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE=true",
"DEFAULT_CATTLE_CATALOG_EXECUTE=true",
"DEFAULT_CATTLE_AUTH_SERVICE_EXECUTE=true",
"DEFAULT_CATTLE_WEBHOOK_SERVICE_EXECUTE=true",
"DEFAULT_CATTLE_SECRETS_API_EXECUTE=true",
"CATTLE_RANCHER_SERVER_IMAGE=rancher/server"
],
"Cmd": [
"/usr/bin/s6-svscan",
"/service"
],
"Image": "rancher/server:v1.4.0",
"Volumes": {
"/var/lib/cattle": {},
"/var/lib/mysql": {},
"/var/log/mysql": {}
},
"WorkingDir": "",
"Entrypoint": [
"/usr/bin/entry"
],
"OnBuild": null,
"Labels": {}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "75040e3b9459631172bcda137560c63a0bcada4ed8bc1187ac59283e0b50f856",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"3306/tcp": null,
"8080/tcp": null
},
"SandboxKey": "/var/run/docker/netns/75040e3b9459",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "db44e4b02038862fdaf621f42729be938174de14cecc1c605ae7f1a5b74f23d1",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "8e97fac0070ecc7418a2dcc2b08ec5bf7a854616b38285e328b6db0bba241022",
"EndpointID": "db44e4b02038862fdaf621f42729be938174de14cecc1c605ae7f1a5b74f23d1",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
}
]
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 25 (2 by maintainers)
@Niceplace @jsalva if you see these errors solely on boot and then they go away, they can be ignored. Here’s what’s happening: In addition to the primary java API server, rancher/server starts up a number of services/processes. One of those is a reverse proxy that sits in front of the java API server (which is actually listening on port 8081) and listens on port 8080. It also starts microservices that make requests to localhost:8080. But, it doesn’t intelligently wait for the reverse proxy to be up before starting the microservices. It instead relies on the fact that those microservices will keep trying to connect until they do so successfully.
Due to the obvious uptick in people reporting this issue, something subtle might have changed recently to cause the reverse proxy to not start as quickly. To alleviate, we might have to add some logic to mitigate this scenario.
From what I can tell in this issue and #7523, most people have a functioning Rancher and are just concerned with the error message.
If that is NOT the case for you and you have an actual functionality problem, could you open a separate issue describing your issue?