gitness: Builds occasionally stuck in the "Building" state with Docker 1.12

In the continuation of https://gitter.im/drone/drone?at=5803905e891a53016314d30d

uname -a

Linux drone 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

commands to start drone server:

docker network create drone-server-net

docker run -d \
    --name=drone-server \
    -e DRONE_SECRET=$DRONE_SECRET \
    -e DRONE_GITHUB=true \
    -e DRONE_GITHUB_CLIENT=$GITHUB_CLIENT \
    -e DRONE_GITHUB_SECRET=$GITHUB_SECRET \
    -e DRONE_ORGS=$GITHUB_ORG \
    -e DRONE_OPEN=true \
    -e DRONE_ADMIN=$GITHUB_ADMIN_USERS \
    -v drone-data:/var/lib/drone \
    --net=drone-server-net \
    -p 80:8000 \
    drone/drone:0.5 server

docker run -d \
    -e DRONE_SECRET=$DRONE_SECRET \
    -e DRONE_SERVER=ws://drone-server:8000/ws/broker \
    -e DRONE_TIMEOUT=15m \
    -e DOCKER_MAX_PROCS=4 \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --net=drone-server-net \
    drone/drone:0.5 agent

YAML:

pipeline:

  build:
    image: node:6.8
    commands:
      - npm i gulp webpack karma --color=always -q

matrix:
  include:
    - axis: 1
    - axis: 2
    - axis: 3
    - axis: 4

When all worker container are exited, some builds occasionally are stuck in “orange” state:


1


2


And containers are not removed, ambassador exit code is 137:

docker ps -a

CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                            PORTS                  NAMES
740c7d3519af        node:6.8             "/bin/sh -c 'echo $DR"   3 minutes ago       Exited (0) About a minute ago                            drone_dUpqOVZnNW4_1
d5a19a6bed0f        plugins/git:latest   "/bin/drone-git"         3 minutes ago       Exited (0) 3 minutes ago                                 drone_dUpqOVZnNW4_0
931454065d90        busybox:latest       "/bin/sleep 86400"       3 minutes ago       Exited (137) About a minute ago                          drone_ambassador_aa49ZRmHlfA
f8e6526d9611        drone/drone:0.5      "/drone agent"           17 minutes ago      Up 17 minutes                     8000/tcp               drunk_ritchie
2d36a428e71a        drone/drone:0.5      "/drone server"          17 minutes ago      Up 17 minutes                     0.0.0.0:80->8000/tcp   drone-server

The issue doesn’t not occur without --color=always (i.e. with no console colors).

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 1
  • Comments: 25

Most upvoted comments

I can confirm this issue with Docker version 1.12.2.

Yep, just checked it. Tasks don’t hang with docker-1.13.0-dev.tgz

I concede: it’s almost certainly that JSON logging issue you’ve found. I suspect the success rate might be due to the amount of log scraping going on for various servers, and the other image not having as many eyes watching it.

Stuck waiting for an engine update now 😕