moby: Docker 1.9.1 hanging at build step "Setting up ca-certificates-java"

A few of us within the office upgraded to the latest version of docker toolbox backed by Docker 1.9.1 and builds are hanging as per the below build output.

docker version:

 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

docker info:

Containers: 10
Images: 57
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 77
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: vbootstrap-vm
ID: LLM6:CASZ:KOD3:646A:XPRK:PIVB:VGJ5:JSDB:ZKAN:OUC4:E2AK:FFTC
Debug mode (server): true
 File Descriptors: 13
 Goroutines: 18
 System Time: 2015-11-24T02:03:35.597772191Z
 EventsListeners: 0
 Init SHA1: 
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

uname -a:

Darwin JRedl-MB-Pro.local 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

Here is a snippet from the docker build uppet that hangs on the Setting up ca-certificates-java line. Something to do with the latest version of docker and openjdk?

update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up ca-certificates-java (20140324) ...

Docker file example:

FROM gcr.io/google_appengine/base

# Prepare the image.
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y -qq --no-install-recommends build-essential wget curl unzip python python-dev php5-mysql php5-cli php5-cgi openjdk-7-jre-headless openssh-client python-openssl && apt-get clean

I can confirm that this is not an issue with Docker 1.9.0 or Docker Toolbox 1.9.0d. Let me know if I can provide any further information but this feels like a regression of some sort within the new version.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 3
  • Comments: 258 (83 by maintainers)

Commits related to this issue

Most upvoted comments

Debian supported this issue.

LATEST QUICK WORKAROUNDS

Distro Workaround
General Use devicemapper/overlay/btrfs (but it may cause another problem…).
If you can upgrade AUFS and build the kernel manually, you can also use AUFS v20160111 or later.
Boot2Docker ✅ Upgrade to v1.10.0 or later
Ubuntu 14.04LTS ✅ Upgrade kernel to 3.13.0-79.123 or later
Ubuntu 15.04 ✅ Upgrade kernel to 3.19.0-51.57 or later
Ubuntu 15.10 ✅ Upgrade kernel to 4.2.0-30.35 or later
Debian 7 ✅ Upgrade kernel to 3.2.73-2+deb7u3 (of linux-image-3.2.0-4-amd64 package) or later
Debian 8 ✅ Upgrade kernel to 3.16.7-ckt20-1+deb8u4 (of linux-image-3.16.0-4-amd64 package) or later
Debian 9 ✅ (does not support AUFS since kernel 3.18-1~exp1)
Gentoo ✅ Upgrade to recent ones (⚠️ not tested)
RHEL/CentOS ✅ (does not support AUFS)
openSUSE ✅ (does not support AUFS)

Distributors Issue Tickets

Distro Status Issue URL
Boot2Docker ✅ Closed https://github.com/boot2docker/boot2docker/pull/1113
Ubuntu ✅ Closed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043
Debian ✅ Closed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812207
+20
AkihiroSuda on Mar 8, 2016

https://github.com/docker/docker/issues/18180#issuecomment-161843456 @andrewgdavis, you’re quite right!

The bug seems to be a regression caused by the commit 296291cd to Linux kernel(mm/filemap.c).

I made a Boot2Docker ISO (boot2docker-v1.9.1-fix1.iso) that omits the commit 296291cd: https://github.com/AkihiroSuda/boot2docker/releases/tag/v1.9.1-fix1

Hope it works for everyone. 😃

The commit 296291cd produces infinite -EINTR loop in mm/filemap.c:generic_perform_write, which fs/aufs/xino.c:do_xino_fwrite() cannot tolerate:

static ssize_t do_xino_fwrite(vfs_writef_t func, struct file *file, void *kbuf,
                  size_t size, loff_t *pos)
{
..
    do {
         /* cannot escape from this loop 
            when func returns -EINTR infinitely! */
        err = func(file, buf.u, size, pos);
    } while (err == -EAGAIN || err == -EINTR);
..
}

As do_xino_fwrite() loops infinitely, zap_pid_ns_processes() (executed in another LWP) cannot return from schedule() when running on a single processor. That’s why we are hitting the bug.

@gilles-duboscq You’re hitting the bug because Canonical originally backported the commit 296291cd to the kernel 3.19 tree: http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/commit/?id=6b08592b8acc677d5b9bb7986343fdd6e0ad3303

+2
AkihiroSuda on Dec 24, 2015

31 participants on this issue and counting.

+2
bean5 on Dec 1, 2015

Updated the table about Ubuntu and Debian.

LATEST QUICK WORKAROUNDS

Distro Workaround
General Use devicemapper/overlay/btrfs (but it may cause another problem…).
If you can upgrade AUFS and build the kernel manually, you can also use AUFS v20160111 or later.
Boot2Docker ✅ Upgrade to v1.10.0 or later
Ubuntu 14.04LTS ✅ Upgrade kernel to 3.13.0-79.123 or later
Ubuntu 15.04 ✅ Upgrade kernel to 3.19.0-51.57 or later
Ubuntu 15.10 ✅ Upgrade kernel to 4.2.0-30.35 or later
Debian 7/8 ⬇️ Downgrade kernel to version 3.16.7-ckt11 of release 3.16.0 or older (dpkg archive by @wzrdtales)
Debian 9 ✅ (does not support AUFS since kernel 3.18-1~exp1)
Gentoo ✅ Upgrade to recent ones (⚠️ not tested)
RHEL/CentOS ✅ (does not support AUFS)
openSUSE ✅ (does not support AUFS)

Distributors Issue Tickets

Distro Status Issue URL
Boot2Docker ✅ Closed https://github.com/boot2docker/boot2docker/pull/1113
Ubuntu ✅ Closed https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043
Debian ◻️ In Progress https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812207

One more thing; I listed up some known bugs about storage drivers: https://github.com/AkihiroSuda/docker-issues

+1
AkihiroSuda on Feb 23, 2016

For information, I listed up some issues and workarounds related to AUFS/Overlay/BtrFS/ZFS/devicemapper storage drivers: https://github.com/AkihiroSuda/docker-issues/

Hope this can help those who are interested in #18180 and others…

+1
AkihiroSuda on Feb 15, 2016

The solution for me was change the storage backend.

Add a line to /etc/default/docker ¡¡BE CAREFUL you’ll lose your container data!!

DOCKER_OPTS="--storage-driver=devicemapper"

I recommend stop docker service, erase docker folder on /var/lib and then add the line and restart docker service.

+1
referup-tarantegui on Feb 1, 2016

I am able to duplicate this issue on my setup (using Docker Machine on Mac).

Here are my findings so far.

As noted by other posters, the simplest way to duplicate this has been to use the boot2docker 1.9.1 ISO with AUFS. This Dockerfile should minimally reproduce the problem fairly quickly:

FROM debian:jessie

ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends openjdk-7-jre-headless

Looking at dmesg, I see some AUFS errors after attempting such a build, but I am not 100% sure they are related:

docker@default:~$ dmesg | tail
aufs au_opts_verify:1597:docker[14186]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:docker[14186]: dirperm1 breaks the protection by the permission bits on the lower branch
aufs au_opts_verify:1597:docker[14186]: dirperm1 breaks the protection by the permission bits on the lower branch
device veth955cc15 entered promiscuous mode
IPv6: ADDRCONF(NETDEV_UP): veth955cc15: link is not ready
eth0: renamed from vethc63e038
IPv6: ADDRCONF(NETDEV_CHANGE): veth955cc15: link becomes ready
docker0: port 2(veth955cc15) entered forwarding state
docker0: port 2(veth955cc15) entered forwarding state
docker0: port 2(veth955cc15) entered forwarding state

If I create a Docker 1.9.1 machine which uses overlay as the storage driver:

$ docker-machine create -d virtualbox --engine-storage-driver overlay overlay

The process does NOT hang and this line runs successfully! Looks like AUFS and/or kernel is the problem.

boot2docker/boot2docker did bump both kernel versions and AUFS commit for the 1.9.1 release, so those are both factors which need to be ruled out or investigated further:

Currently trying 1.9.0 ISO with a 1.9.1 binary to see if the surface area of potential bug area can be reduced further.

+1
nathanleclaire on Dec 1, 2015

@bean5 please keep your comments constructive

+1
thaJeztah on Dec 1, 2015
Read more comments on GitHub
← moby: docker 19.03-RC3: cp broken with debian containers (arm?)
moby: docker 20.10.4 on linux and windows breaks docker-compose run -T →