mainflux: docker-compose fails

docker-compose us is running out-of-order dockers.

Without a backoff and re-try with timeour in services themselves, we can forget about using docker-compose and we must fallback to previous starting script.

drasko@Marx:~/go/src/github.com/mainflux/mainflux/docker$ docker-compose up
Creating mainflux-manager ... 
Creating mainflux-nats ... 
Creating mainflux-postgres ... 
Creating mainflux-normalizer ... 
Creating mainflux-manager
Creating mainflux-nginx ... 
Creating mainflux-nats
Creating mainflux-normalizer
Creating mainflux-nginx
Creating mainflux-nats ... done
Creating mainflux-http ... 
Creating mainflux-mqtt ... 
Creating mainflux-mqtt
Creating mainflux-mqtt ... done
Attaching to mainflux-manager, mainflux-nats, mainflux-normalizer, mainflux-postgres, mainflux-nginx, mainflux-http, mainflux-mqtt
mainflux-manager | {"error":"dial tcp: lookup postgres on 10.20.0.1:53: no such host","ts":"2018-03-13T02:17:55.877664432Z"}
mainflux-nats   | [1] 2018/03/13 02:17:56.008535 [INF] Starting nats-server version 1.0.2
mainflux-nats   | [1] 2018/03/13 02:17:56.008625 [INF] Starting http monitor on 0.0.0.0:8222
mainflux-normalizer | {"error":"Failed to connect: nats: no servers available for connection","ts":"2018-03-13T02:17:56.16486139Z"}
mainflux-nats   | [1] 2018/03/13 02:17:56.008662 [INF] Listening for client connections on 0.0.0.0:4222
mainflux-nats   | [1] 2018/03/13 02:17:56.008666 [INF] Server is ready
mainflux-postgres | The files belonging to this database system will be owned by user "postgres".
mainflux-postgres | This user must also own the server process.
mainflux-postgres | 
mainflux-nats   | [1] 2018/03/13 02:17:56.134986 [INF] Listening for route connections on 0.0.0.0:6222
mainflux-nginx  | 2018/03/13 02:17:56 [emerg] 1#1: host not found in upstream "mainflux-manager:8180" in /etc/nginx/nginx.conf:58
mainflux-postgres | The database cluster will be initialized with locale "en_US.utf8".
mainflux-http   | {"error":"nats: no servers available for connection","ts":"2018-03-13T02:17:56.689929955Z"}
mainflux-nginx  | nginx: [emerg] host not found in upstream "mainflux-manager:8180" in /etc/nginx/nginx.conf:58
mainflux-postgres | The default database encoding has accordingly been set to "UTF8".
mainflux-postgres | The default text search configuration will be set to "english".
mainflux-postgres | 
mainflux-postgres | Data page checksums are disabled.
mainflux-postgres | 
mainflux-manager exited with code 1
mainflux-postgres | fixing permissions on existing directory /var/lib/postgresql/data ... ok
mainflux-postgres | creating subdirectories ... ok
mainflux-postgres | selecting default max_connections ... 100
mainflux-postgres | selecting default shared_buffers ... 128MB
mainflux-postgres | selecting dynamic shared memory implementation ... posix
mainflux-postgres | creating configuration files ... ok
mainflux-normalizer exited with code 1
mainflux-postgres | running bootstrap script ... ok
mainflux-postgres | performing post-bootstrap initialization ... sh: locale: not found
mainflux-postgres | 2018-03-13 02:17:56.543 UTC [29] WARNING:  no usable system locales were found
mainflux-postgres | ok
mainflux-postgres | syncing data to disk ... ok
mainflux-postgres | 
mainflux-postgres | Success. You can now start the database server using:
mainflux-postgres | 
mainflux-postgres |     pg_ctl -D /var/lib/postgresql/data -l logfile start
mainflux-postgres | 
mainflux-postgres | 
mainflux-postgres | WARNING: enabling "trust" authentication for local connections
mainflux-postgres | You can change this by editing pg_hba.conf or using the option -A, or
mainflux-postgres | --auth-local and --auth-host, the next time you run initdb.
mainflux-postgres | waiting for server to start....2018-03-13 02:17:56.858 UTC [34] LOG:  listening on IPv4 address "127.0.0.1", port 5432
mainflux-postgres | 2018-03-13 02:17:56.858 UTC [34] LOG:  could not bind IPv6 address "::1": Address not available
mainflux-postgres | 2018-03-13 02:17:56.858 UTC [34] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
mainflux-postgres | 2018-03-13 02:17:56.865 UTC [34] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
mainflux-postgres | 2018-03-13 02:17:56.883 UTC [35] LOG:  database system was shut down at 2018-03-13 02:17:56 UTC
mainflux-postgres | 2018-03-13 02:17:56.887 UTC [34] LOG:  database system is ready to accept connections
mainflux-postgres |  done
mainflux-postgres | server started
mainflux-mqtt   | {"pid":1,"hostname":"2bc75be03e01","level":30,"time":1520907476986,"msg":"listening","address":"::","family":"IPv6","port":8880,"protocol":"http","v":1}
mainflux-mqtt   | {"pid":1,"hostname":"2bc75be03e01","level":30,"time":1520907476988,"msg":"listening","address":"::","family":"IPv6","port":1883,"protocol":"tcp","v":1}
mainflux-mqtt   | events.js:165
mainflux-mqtt   |       throw err;
mainflux-mqtt   |       ^
mainflux-mqtt   | 
mainflux-mqtt   | Error: Uncaught, unspecified "error" event. (Could not connect to server: Error: getaddrinfo ENOTFOUND nats nats:4222)
mainflux-mqtt   |     at Client.emit (events.js:163:17)
mainflux-mqtt   |     at Socket.<anonymous> (/node_modules/nats/lib/nats.js:405:14)
mainflux-mqtt   |     at emitOne (events.js:96:13)
mainflux-mqtt   |     at Socket.emit (events.js:188:7)
mainflux-mqtt   |     at connectErrorNT (net.js:1025:8)
mainflux-mqtt   |     at _combinedTickCallback (internal/process/next_tick.js:80:11)
mainflux-mqtt   |     at process._tickCallback (internal/process/next_tick.js:104:9)
mainflux-nginx exited with code 1
mainflux-http exited with code 1
mainflux-postgres | CREATE DATABASE
mainflux-postgres | 
mainflux-postgres | CREATE ROLE
mainflux-postgres | 
mainflux-postgres | 
mainflux-postgres | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
mainflux-postgres | 
mainflux-postgres | waiting for server to shut down....2018-03-13 02:17:57.224 UTC [34] LOG:  received fast shutdown request
mainflux-postgres | 2018-03-13 02:17:57.228 UTC [34] LOG:  aborting any active transactions
mainflux-postgres | 2018-03-13 02:17:57.229 UTC [34] LOG:  worker process: logical replication launcher (PID 41) exited with exit code 1
mainflux-postgres | 2018-03-13 02:17:57.229 UTC [36] LOG:  shutting down
mainflux-postgres | 2018-03-13 02:17:57.262 UTC [34] LOG:  database system is shut down
mainflux-postgres |  done
mainflux-postgres | server stopped
mainflux-postgres | 
mainflux-postgres | PostgreSQL init process complete; ready for start up.
mainflux-postgres | 
mainflux-postgres | 2018-03-13 02:17:57.335 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
mainflux-postgres | 2018-03-13 02:17:57.335 UTC [1] LOG:  listening on IPv6 address "::", port 5432
mainflux-postgres | 2018-03-13 02:17:57.343 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
mainflux-postgres | 2018-03-13 02:17:57.370 UTC [47] LOG:  database system was shut down at 2018-03-13 02:17:57 UTC
mainflux-postgres | 2018-03-13 02:17:57.378 UTC [1] LOG:  database system is ready to accept connections
mainflux-mqtt exited with code 1

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 27 (14 by maintainers)

Commits related to this issue

NOISSUE - Fix time unit in charts (use milliseconds) (#179) * NOISSUE - Fix time unit in charts (use milliseconds) Signed-off-by: Manuel Imperiale <manuel.imperiale@gmail.com> * Add toMillisecs... — committed to mteodor/mainflux by manuio 3 years ago

Most upvoted comments

Hi,

I tested the docker composition and for most cases with decent network connection it will work with the default docker-compose parameters. I also did few tests running GNU/Linux from a USB Stick (a bit extreme 😛, but I wanted to test with slower hardware/network connection) and with the default docker-compose parameters (COMPOSE_HTTP_TIMEOUT=60) and I managed to get the composition fail:

....
....
....
ERROR: for mainflux-nats  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for mainflux-postgres  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for mainflux-manager  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for mainflux-normalizer  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for mainflux-nginx  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for nats  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for postgres  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for manager  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for normalizer  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for nginx  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Than I deleted the containers and the images and started the composition with:

$ COMPOSE_HTTP_TIMEOUT=300 docker-compose up

Everything went well and Mainflux started properly.

I think maybe we could add few comments in the README.md and the getting-started.md that in case of timeouts the users should raise the value of the COMPOSE_HTTP_TIMEOUT envvar. On the other hand, that already comes as an error from docker-compose, so If the users follow the logs they can get Mainflux up and running.

If we want ot be nice we could add a troubleshooting section in our docs, but I don’t think that this is a must, because if the users follow the logs they can solve all the issues with the timings.

Best Regards, Jovan

chombium on Mar 19, 2018