concourse: 3.11.0 worker restart regression

Bug Report

  • Concourse version: 3.11.0
  • Deployment type (BOSH/Docker/binary): docker
  • Infrastructure/IaaS: Ubuntu 16.04 VM
  • Did this used to work? yes in 3.10.0

When restarting a worker (or rebooting the machine), 3.11.0 workers fail to start if the volumes.img already exists. Removing the file allows the worker to start, but fails on next restart/reboot.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 7
  • Comments: 16 (1 by maintainers)

Commits related to this issue

Most upvoted comments

I am facing same issue with concourse helm chart with image version 7.7.I had deployed previously with baggageclaim driver set to : naive.But it did not work since all went down.Again I redployed the total helm via brtfs & making mkfs.brtfs flag as enabled setting the required binary,but still facing the same issue

@billimek unfortunately it doesn’t seem to work.

We have the same problem with Concourse 5.0 and the following versions:

# docker -v
Docker version 18.06.1-ce, build e68fc7a
# docker-compose -v
docker-compose version 1.23.2, build 1110ad01
# uname -a
Linux seazoet3 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

How to reproduce:

  1. Use the official docker-compose at https://concourse-ci.org/docker-compose.yml
  2. Start with docker-compose up
  3. Stop with Ctrl-C
  4. Try to start again with docker-compose up and you get the following error:
concourse_1     | {"timestamp":"2019-03-19T13:26:25.966851698Z","level":"info","source":"atc","message":"atc.cmd.start","data":{"session":"1"}}
concourse_1     | {"timestamp":"2019-03-19T13:26:26.019497356Z","level":"info","source":"atc","message":"atc.cmd.finish","data":{"duration":122126,"session":"1"}}
concourse_1     | {"timestamp":"2019-03-19T13:26:26.246555949Z","level":"error","source":"baggageclaim","message":"baggageclaim.fs.run-command.failed","data":{"args":["bash","-e","-x","-c","\n\t\tif [ ! -e $IMAGE_PATH ] || [ \"$(stat --printf=\"%s\" $IMAGE_PATH)\" != \"$SIZE_IN_BYTES\" ]; then\n\t\t\ttouch $IMAGE_PATH\n\t\t\ttruncate -s ${SIZE_IN_BYTES} $IMAGE_PATH\n\t\tfi\n\n\t\tlo=\"$(losetup -j $IMAGE_PATH | cut -d':' -f1)\"\n\t\tif [ -z \"$lo\" ]; then\n\t\t\tlo=\"$(losetup -f --show $IMAGE_PATH)\"\n\t\tfi\n\n\t\tif ! file $IMAGE_PATH | grep BTRFS; then\n\t\t\tmkfs.btrfs --nodiscard $IMAGE_PATH\n\t\tfi\n\n\t\tmkdir -p $MOUNT_PATH\n\n\t\tif ! mountpoint -q $MOUNT_PATH; then\n\t\t\tmount -t btrfs $lo $MOUNT_PATH\n\t\tfi\n\t"],"command":"/bin/bash","env":["PATH=/usr/local/concourse/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","MOUNT_PATH=/worker-state/volumes","IMAGE_PATH=/worker-state/volumes.img","SIZE_IN_BYTES=200101810176"],"error":"exit status 1","session":"3.1","stderr":"+ '[' '!' -e /worker-state/volumes.img ']'\n++ stat --printf=%s /worker-state/volumes.img\n+ '[' 200101810176 '!=' 200101810176 ']'\n++ cut -d: -f1\n++ losetup -j /worker-state/volumes.img\n+ lo=/dev/loop13\n+ '[' -z /dev/loop13 ']'\n+ file /worker-state/volumes.img\nbash: line 11: file: command not found\n+ grep BTRFS\n+ mkfs.btrfs --nodiscard /worker-state/volumes.img\n/worker-state/volumes.img appears to contain an existing filesystem (btrfs).\nERROR: use the -f option to force overwrite of /worker-state/volumes.img\n","stdout":"btrfs-progs v4.15.1\nSee http://btrfs.wiki.kernel.org for more information.\n\n"}}
concourse_1     | {"timestamp":"2019-03-19T13:26:26.246649222Z","level":"error","source":"baggageclaim","message":"baggageclaim.failed-to-set-up-driver","data":{"error":"failed to create btrfs filesystem: exit status 1"}}
concourse_1     | failed to create btrfs filesystem: exit status 1
concourse_concourse_1 exited with code 1

I think this error is actually going to be fixed by https://github.com/concourse/concourse-docker/pull/15

Removing the volumes.img was probably just masking the check using the missing file command. Can’t wait to see this PR included in a release!

Got the same error each time the worker restart with this configuration :

  • Concourse version : 3.13.0
  • Deployment type (BOSH/Docker/binary): docker
  • Infrastructure:IaaS : Mint 18.3