moby: Proposal: Containers should not be considered started until the TCP ports they expose are open

Review with @crosbymichael @LK4D4 @duglin @diogomonica @calavera @cpuguy83 @vishh @ewindisch

It feels like this functionality should not be Docker’s responsibility: there will never be enough use cases to support (TCP, UDP, why not HTTP?) and it would bring more technical complexity, when really it should be the caller’s responsibility to make sure the container is “ready” as opposed to “started”. This would rather be orchestration tools’ responsibility.

Health checks are a great use case, but again, there seems to be consensus on the fact that they are the orchestration tools’ responsibility. Ping @mavenugo what do you think?

Also there was an argument in favor, stating that docker users and docker image authors are separate people, and although one could expect from image authors to better handle failure to establish a handshake, one cannot expect that from docker users that simply want to run existing applications that aren’t as failsafe. The counterargument is that there is no way to make failsafe, an application that is not. If a container is considered “started” when its tcp endpoint is listening, just so that all another application has to do is docker wait on a container, then if for some reason the container is not even able to create the tcp socket, the application that’s docker waiting will fail anyway. So if you need network guarantees, it makes sense to decouple that from Docker and have that in a separate tool.

@aanand @bfirsh @vieux @aluzzardi @mavenugo what do you guys think? Could compose or swarm provide health checks and some guarantees?

I’m closing this, but feel free to continue the discussion. We will reconsider if needed.

But what about UDP? What about applications, which open the socket very fast, but start to provide a service a bit later? (I’m pretty sure somebody will have such case)

What we talk here about is more or less a functionality of an orchestrator which manages software deployments:

• start SW deployment
• wait till SW is deployed and services are started
• continue with next steps (e.g. next SW deployment on the next instance)

We have a bit another picture with docker&microservices: you instantiate containers from image(s) and have some kind of initialization phase (which takes a while, otherwise we wouldn’t have this discussion). But you don’t have this classical SW deployment here, as its usually part of building the images. So this issue is about to have similar behaviour within docker, it might be a feature for fig/compose (docker/compose#374) (with open questions regarding reliable health checks not only on the TCP layer, but higher layers as well) , which implements the abstraction layer around hypervisor. but IMHO not for hypervisor itself.

I deal with this problem like this (simple TCP check): Dockerfile:

...
ENV START_CMD="some daemon"
ENV MYSQL_HOST="mysql"
ENV MYSQL_PORT="3306"
...
CMD /etc/docker/start
...

/etc/docker/start:

#!/bin/sh
# Wait for database to get available

MYSQL_LOOPS="10"

#wait for mysql
i=0
while ! nc $MYSQL_HOST $MYSQL_PORT >/dev/null 2>&1 < /dev/null; do
  i=`expr $i + 1`
  if [ $i -ge $MYSQL_LOOPS ]; then
    echo "$(date) - ${MYSQL_HOST}:${MYSQL_PORT} still not reachable, giving up"
    exit 1
  fi
  echo "$(date) - waiting for ${MYSQL_HOST}:${MYSQL_PORT}..."
  sleep 1
done

#continue with further steps

#start the daemon
exec $START_CMD

👍 Sort of defeats purpose of orchestration, if containers are started before containers it depends on are ready.

in case anyone else arrives here looking to find a way to ensure postgres connection is open before proceeding with a webapp… here is what I landed on and works nicely. (Add postgresql-client to the webapp Dockerfile)

#!/bin/sh
su -c "
    while ! psql --host=webappservicedb --username=postgres > /dev/null 2>&1; do
        echo 'Waiting for webappservicedb connection with postgres...'
        sleep 1;
    done;
    echo 'Connected to postgres...';"
su -c "python manage.py migrate"
su -c "python manage.py runserver 0.0.0.0:8080"

I don’t feel this should be part of Docker itself especially as ports may or not be open depending on whatever conditions. This should be part of whatever is handling the startup of containers if they are meant to be run as a web service (e.g. Kubernetes).

decking has an option ready for doing this.

Having compose allow you to define conditions that must be met before that service is considered started would be cool. It does sounds like docker itself may be the wrong place for this, should we create an issue in compose?

On Wed, May 13, 2015 at 7:09 AM, Günter Zöchbauer notifications@github.com wrote:

No idea if this can be done, but I think it would be nice getting an event through the GET /events endpoint when a TCP port is being bound to, with the need to opt-in for each port when a container is created.

— Reply to this email directly or view it on GitHub https://github.com/docker/docker/issues/7445#issuecomment-101642931.

Sincerely,

Colton Leekley-Winslow 651-242-9559 lwcolton@gmail.com