moby: Starting containers should not be considered for load balancing on service upscaling

Output of docker version:

Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64
 Experimental: true

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

Containers: 58
 Running: 3
 Paused: 0
 Stopped: 55
Images: 41
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-8:1-524491-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.275 GB
 Data Space Total: 107.4 GB
 Data Space Available: 106.1 GB
 Metadata Space Used: 4.481 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.143 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: active
 NodeID: REDACTED
 Is Manager: true
 ClusterID: REDACTED
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: REDACTED
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.671 GiB
Name: REDACTED
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 58
 Goroutines: 162
 System Time: REDACTED
 EventsListeners: 3
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: true
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.): VPS (KVM)

Steps to reproduce the issue:

1. Create a Dockerfile for a slow-starting container image, e.g. with sleep (or a Java + Spring application), with a running HTTP server, exposed port, and health check (picked dockercloud/hello-world here):

   FROM dockercloud/hello-world
   
   RUN apk add --update curl && \
       rm -rf /var/cache/apk/*
   
   HEALTHCHECK --interval=1m --timeout=3s \
       CMD curl -f http://127.0.0.1:80/ || exit 1
   
   CMD sleep 20 && php-fpm -d variables_order="EGPCS" && (tail -F /var/log/nginx/access.log &) && exec nginx -g "daemon off;"
   

   …and create the slow-container image:

   # docker build -t slow-container .
   

2. Create a slow-service:

   # docker service create --publish 8080:80 --name slow-service --replicas=1 slow-container
   ev070x9sh8el7frmqw57v97ts
   

   You may want to ensure that the health check is working properly, which indicates container as starting first:

   # docker ps
   CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS                            PORTS               NAMES
   91cc5a672b24        slow-container:latest                       "/bin/sh -c 'sleep 20"   2 seconds ago       Up 1 seconds (health: starting)   80/tcp              slow-service.1.cqupqtrfeiw9xyhumr6q5za6w
   

   The status becomes healthy after some time:

   # docker ps
   CONTAINER ID        IMAGE                                       COMMAND                  CREATED             STATUS                    PORTS               NAMES
   91cc5a672b24        slow-container:latest                       "/bin/sh -c 'sleep 20"   2 minutes ago       Up 2 minutes (healthy)    80/tcp              slow-service.1.cqupqtrfeiw9xyhumr6q5za6w
   

3. Ensure service is responding properly to HTTP requests:

   # while true; do curl -s --show-error -I http://127.0.0.1:8080 | head -n 1; sleep 1; done
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   ^C
   

4. Scale the service to two nodes:

   # docker service scale slow-service=2
   slow-service scaled to 2
   

   Ensure the new container is in starting state (# docker ps) and run the HTTP check against the slow-service again:

   # while true; do curl -s --show-error -I http://127.0.0.1:8080 | head -n 1; sleep 1; done
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   curl: (7) Failed connect to 127.0.0.1:8080; Connection refused
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   HTTP/1.1 200 OK
   ^C
   

   Finally, ensure the new container is healthy now (# docker ps).

Describe the results you received:

Obviously, the new container (created due to upscaling) was considered for load balancing immediately after it was created. This behavior causes failures of every 2nd connection to the service on consumer side.

Accordingly, upscaling from 1 to 5 tasks would cause failures in 4 of 5 requests.

Describe the results you expected:

When HEALTHCHECK is defined in Dockerfile, new containers should not participate on load balancing as long as their status is starting.

Additional information you deem important (e.g. issue happens only occasionally):

The issue can be reproduced on every upscaling. Might be related to #25464.