balena-engine: Error: layers from manifest don't match image configuration

Description

Image download fails with error:

Failed to download image 'registry2.balena-cloud.com/v2/9bc400dcc9a75bc3be299a20d38e9c76@sha256:1709f3810eaecf4817bd419c2228b8218d4538fa1d208c4915cc0cde49282cc9' due to 'layers from manifest don't match image configuration'

Output of balena-engine version:

Client:
 Version:           19.03.13-dev
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        074a481789174b4b6fd2d706086e8ffceb72e924
 Built:             Sun Aug 16 10:33:45 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server:
 Engine:
  Version:          19.03.13-dev
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       074a481789174b4b6fd2d706086e8ffceb72e924
  Built:            Sun Aug 16 10:33:45 2020
  OS/Arch:          linux/arm
  Experimental:     true
 containerd:
  Version:          1.2.0+unknown
  GitCommit:        
 runc:
  Version:          
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

Output of balena-engine info:

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 19.03.13-dev
 Storage Driver: aufs
  Root Dir: /var/lib/docker/aufs
  Backing Filesystem: extfs
  Dirs: 37
  Dirperm1 Supported: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm: 
  NodeID: 
  Is Manager: false
  Node Address: 
 Runtimes: bare runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version: 
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: 949e6fa-dirty (expected: fec3683b971d9)
 Kernel Version: 4.19.118
 Operating System: balenaOS 2.58.3+rev1
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 745.6MiB
 Name: 81e8074
 ID: X4GL:EF4Y:PES6:2IP6:K2HI:T67S:6XJX:7XP3:FEYW:GTJZ:NUGM:3XCU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

Additional environment details (device type, OS, etc.): Device type: Raspberry Pi 3 OS version: balenaOS 2.58.3+rev1

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Comments: 34 (13 by maintainers)

Most upvoted comments

I encountered a device on support where this error message was "manifest"ing (excuse the pun) while a device was updating between releases A & B. I tried an experiment which happened to resolve, which did not involve rebooting or restarting anything. Below is the steps I took; hopefully it’ll aid in future investigations of this issue:

  1. Stop service & remove base image from release A (i.e. source image, not target image) of service that was erroring when Supervisor was pulling delta
  2. Manually pull the base image directly from balena registry
  3. Restart Supervisor, wait for SV to attempt delta pull to release B again
  4. Observe that delta pull was successful this time without manifesting the error

The user pinned to 4 different target releases in a short time frame – could it be that the base image manifest got corrupt from multiple attempts from the Engine to interact with it (race condition), due to the 4 pins in quick succession?

@cywang117 The solution you provided did resolve the issue for the customer, reporting as requested.

Moving into here some debugging notes I was keeping elsewhere.

Findings after working on ticket (1)

  1. We can confirm this is not a problem with the delta image itself. Everything makes me believe this is some corruption (caused by a bug, not by bad SD card or anything) on the Engine persistent state.
  2. The error is being triggered here.
  3. In this case, the only mismatch was in the first layer:
  • configRootFS.DiffIDs[0] == sha256:aaaaaaa (this is coming from the io.resin.delta.config label)
  • downloadedRootFS.DiffIDs[0] == sha256:bbbbbbb (which seems to be ID of the delta layer itself, before it is applied).
  1. How we got into this state is not clear (and is the key to understand the issue).
  2. On the device, I found this file: /var/lib/docker/image/overlay2/distribution/v2metadata-by-diffid/sha256/bbbbbbb, which I wasn’t expecting to be there – I don’t see it when downloading the base and then the delta in another device. (Again, this is pointing to the delta layer itself.) Not sure how (or if) this is relevant, but it may be part of the corruption we are looking for.

Trying to form an hypothesis from the findings above

Maybe this is happening:

  • We download a delta layer as part of the app update.
    • Normally, we’d apply the delta immediately, that’s the expected Engine behavior (pull delta == apply delta).
    • But, for some reason, applying the delta somehow fails.
    • We end up with the delta layer on disk. On a normal delta pull, we’d not have the delta layer on disk, but on one affected device the layer was found on disk.
  • This would match the logs in the case I analyzed above, AFAIU: the real image metadata (from the io.resin.delta.config label points to the target layer, but we are finding the delta layer instead.
  • Any subsequent attempts to repeat the update would keep failing with the same error.
    • Because if the delta layer is already found on disk, we’ll not try to pull it again. And if we don’t pull it again, the delta never gets applied.
  • This would also explain why only some devices of a fleet are affected.

Findings after working on ticket (2)

Largely similar to ticket (1): failing at the same point, with a mismatch in only one of the layers. Some major differences, though:

  • The image causing error was not a delta.
    • This invalidates my previous hypothesis.
  • Manually pulling the image still solved the issue.
    • Which begs the question: wasn’t this exactly the same thing the Supervisor was trying to do? Why did the manual pull work and the Supervisor pull didn’t? (And I think this question is relevant also for the delta case.)