balena-cli: Preload fails with "error initializing graphdriver: driver not supported"

  • Cli version: 9.12.2
  • Standalone install: no
  • Node version: 11.1.0
  • Npm version: 6.4.1
  • Operating system and architecture: macOS 10.14.3
  • Docker version: 18.09.1 stable (Docker Desktop for Mac)

First off, the preloader is an awesome tool and I really appreciate it! I have used it successfully in a Linux environment, but under macOS it fails (using the Intel NUC base image https://files.resin.io/resinos/intel-nuc/2.29.2%2Brev2.prod/image/balena.img.zip)

$ balena preload balena.img
Building Docker preloader image. [========================] 100%



/ Creating preloader container
\ Starting preloader container
| Reading image information
1: Step 1/7 : FROM docker:17.10.0-ce-dind
 ---> 9769e0f3f9cb
Step 2/7 : RUN apk update && apk add --no-cache python3 parted btrfs-progs docker util-linux sfdisk file coreutils sgdisk
 ---> Using cache
 ---> 8e051386d0ca
Step 3/7 : COPY ./requirements.txt /tmp/
 ---> Using cache
 ---> acbb5071e45c
Step 4/7 : RUN pip3 install -r /tmp/requirements.txt
 ---> Using cache
 ---> bf95343e0281
Step 5/7 : COPY ./src /usr/src/app
 ---> Using cache
 ---> 1a0ed03b49a9
Step 6/7 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 334af6145047
Step 7/7 : CMD ["python3", "/usr/src/app/preload.py"]
 ---> Using cache
 ---> 510a76c6dcde
Successfully built 510a76c6dcde
Successfully tagged balena/balena-preload:latest
Waiting for Docker to start...
Exception in thread background thread for pid 209:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/site-packages/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/sh.py", line 2459, in background_thread
    handle_exit_code(exit_code)
  File "/usr/lib/python3.6/site-packages/sh.py", line 2157, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/usr/lib/python3.6/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /usr/local/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmp5yk6hne2/docker --host=tcp://0.0.0.0:64730

  STDOUT:


  STDERR:
time="2019-02-13T16:29:32.361054479Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2019-02-13T16:29:32.367081076Z" level=info msg="libcontainerd: new containerd process, pid: 227"
Error starting daemon: error initializing graphdriver: driver not supported


Traceback (most recent call last):
  File "/usr/src/app/preload.py", line 825, in <module>
    result = method(**data.get("parameters", {}))
  File "/usr/src/app/preload.py", line 785, in get_image_info
    images, supervisor_version = get_images_and_supervisor_version()
  File "/usr/src/app/preload.py", line 668, in get_images_and_supervisor_version
    return _get_images_and_supervisor_version(inner_image_path)
  File "/usr/src/app/preload.py", line 644, in _get_images_and_supervisor_version
    with docker_context_manager(driver, mountpoint):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/src/app/preload.py", line 511, in docker_context_manager
    running_dockerd = start_docker_daemon(storage_driver, docker_dir)
  File "/usr/src/app/preload.py", line 480, in start_docker_daemon
    running_dockerd.wait()
  File "/usr/lib/python3.6/site-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python3.6/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /usr/local/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmp5yk6hne2/docker --host=tcp://0.0.0.0:64730

  STDOUT:


  STDERR:
time="2019-02-13T16:29:32.361054479Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2019-02-13T16:29:32.367081076Z" level=info msg="libcontainerd: new containerd process, pid: 227"
Error starting daemon: error initializing graphdriver: driver not supported

If you need help, don't hesitate in contacting us at:

  GitHub: https://github.com/balena-io/balena-cli/issues/new
  Forums: https://forums.balena.io

Thanks for your help!

Front logo Front conversations

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 3
  • Comments: 33 (14 by maintainers)

Most upvoted comments

@pdcastro I’ve just encountered this issue here too. On latest macOS and latest Docker Desktop, the CLI preload command just hangs at Reading image information - is there a way to at least catch the error and inform the CLI process, so users aren’t left waiting and wondering what’s going on?

To clarify, this is because we still use AUFS on a bunch of balenaOS devices. Preloading OS images that have the engine with overlay2 used for containers should not result in this problem. And as soon as we migrate from aufs to overlay2 this will be resolved. In fact, we could also move preload operations to the cloud so that end-users don’t need to care.

@panbanda, thanks for asking. I have now updated an earlier comment to advise using a Linux VM and the balena CLI for Linux instead of downgrading Docker, because those older versions of Docker are getting just too old. Have a look at the updated comment for some additional details.

Balena is still working towards replacing AUFS with overlay2 in affected balenaOS images, and the tricky process of reliably migrating devices in the field from AUFS to overlay2 as part of host OS updates. Progress is a bit slow but it is critical that the implementation is done right, so it is not something being rushed.

This is great news! Is there a public kanban or something to view the progress / timelines of the different boards? I’m really only needing the rpi0 but I’m sure others would be interested as well.

Some info here from the forums that is super helpful:

the aufs to overlay migration takes place with balenaOS 2.84.0, so it will happen as device types get updated. The engine will then migrate applications from aufs to overlay avoiding the need to re-download containers.

The affected device types are:

beaglebone-black intel-edison intel-nuc odroid-c1 raspberry-pi raspberry-pi2 raspberrypi3 raspberrypi3-64 fincm3 revpi-core-3 npe-x500-m3 up-board

This issue is caused by the removal of the AUFS storage driver (--storage-driver=aufs) in Docker Community Edition 2.0.0.0-mac78 2018-11-19 and later versions, including Docker Desktop for macOS and Windows.

Known Workarounds

Last edit: Nov 2021

Use Linux (e.g. Ubuntu) in a virtual machine like Ubuntu Multipass or VirtualBox, and install the balena CLI for Linux in the virtual machine. When doing so, do not place the OS image file (to be preloaded) in “shared folders” (network mounted filesystem) because the preload implementation uses losetup behind the scenes to mount images within images, which does not work well with a network filesystem. Instead, copy the image file to the virtual machine’s own filesystem. You can of course use shared folders in order to copy files to/from the VM’s own filesystem before/after preloading.

Back in 2019, another workaround was to downgrade to Docker Community Edition 18.06.1-ce-mac73 2018-08-29 (which ships with Docker 18.06.1-ce). However, those older versions of Docker are now “really old” and offer poor support for modern features like multiarch images, so many users will now prefer to use a Linux VM.

Proper Solution

Balena is working towards replacing AUFS with overlay2 in affected balenaOS images. Newer device types like the Raspberry Pi 4 already use overlay2. For other device types, the difficulty currently being addressed is the host OS update of devices in the field running old balenaOS releases that still use AUFS, and need to be migrated to overlay2 as part of the host OS update.

is there a way to at least catch the error and inform the CLI process, so users aren’t left waiting and wondering what’s going on?

@chrisys, @kenna-smith, @sazerzac, @erlend, @briggySmalls, (cc: @klutchell) thanks for the feedback. FYI, the following CLI releases improved on error reporting such that the error is caught and reported straight away:

  • v12.44.13 (2021-05-07)
  • v12.44.22 (2021-06-24)
  • v12.9.5 (2020-07-28)

The latest CLI release has the best error handling capabilities.

@shawaj, thanks for the heads up on the broken links for the old Docker releases. 👍   I’ll fix earlier comments of mine.

The ultimate solution to this issue is to replace AUFS with overlay2 in affected balenaOS images, and balena is making good progress in this direction. I understand that the difficulty currently being addressed is the host OS update of devices in the field where installed balenaOS images use AUFS and the new balenaOS images use overlay2. The image will be converted as part of the host OS update.