balena-engine: Error: layers from manifest don't match image configuration
Description
Image download fails with error:
Failed to download image 'registry2.balena-cloud.com/v2/9bc400dcc9a75bc3be299a20d38e9c76@sha256:1709f3810eaecf4817bd419c2228b8218d4538fa1d208c4915cc0cde49282cc9' due to 'layers from manifest don't match image configuration'
Output of balena-engine version:
Client:
Version: 19.03.13-dev
API version: 1.40
Go version: go1.12.17
Git commit: 074a481789174b4b6fd2d706086e8ffceb72e924
Built: Sun Aug 16 10:33:45 2020
OS/Arch: linux/arm
Experimental: false
Server:
Engine:
Version: 19.03.13-dev
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: 074a481789174b4b6fd2d706086e8ffceb72e924
Built: Sun Aug 16 10:33:45 2020
OS/Arch: linux/arm
Experimental: true
containerd:
Version: 1.2.0+unknown
GitCommit:
runc:
Version:
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
balena-engine-init:
Version: 0.13.0
GitCommit: 949e6fa-dirty
Output of balena-engine info:
Client:
Debug Mode: false
Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 3
Server Version: 19.03.13-dev
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 37
Dirperm1 Supported: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host null
Log: journald json-file local
Swarm:
NodeID:
Is Manager: false
Node Address:
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-engine-init
containerd version:
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: 949e6fa-dirty (expected: fec3683b971d9)
Kernel Version: 4.19.118
Operating System: balenaOS 2.58.3+rev1
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 745.6MiB
Name: 81e8074
ID: X4GL:EF4Y:PES6:2IP6:K2HI:T67S:6XJX:7XP3:FEYW:GTJZ:NUGM:3XCU
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.
Additional environment details (device type, OS, etc.): Device type: Raspberry Pi 3 OS version: balenaOS 2.58.3+rev1
About this issue
- Original URL
- State: open
- Created 6 years ago
- Comments: 34 (13 by maintainers)
I encountered a device on support where this error message was "manifest"ing (excuse the pun) while a device was updating between releases A & B. I tried an experiment which happened to resolve, which did not involve rebooting or restarting anything. Below is the steps I took; hopefully it’ll aid in future investigations of this issue:
The user pinned to 4 different target releases in a short time frame – could it be that the base image manifest got corrupt from multiple attempts from the Engine to interact with it (race condition), due to the 4 pins in quick succession?
@cywang117 The solution you provided did resolve the issue for the customer, reporting as requested.
Moving into here some debugging notes I was keeping elsewhere.
Findings after working on ticket (1)
configRootFS.DiffIDs[0] == sha256:aaaaaaa(this is coming from theio.resin.delta.configlabel)downloadedRootFS.DiffIDs[0] == sha256:bbbbbbb(which seems to be ID of the delta layer itself, before it is applied)./var/lib/docker/image/overlay2/distribution/v2metadata-by-diffid/sha256/bbbbbbb, which I wasn’t expecting to be there – I don’t see it when downloading the base and then the delta in another device. (Again, this is pointing to the delta layer itself.) Not sure how (or if) this is relevant, but it may be part of the corruption we are looking for.Trying to form an hypothesis from the findings above
Maybe this is happening:
io.resin.delta.configlabel points to the target layer, but we are finding the delta layer instead.Findings after working on ticket (2)
Largely similar to ticket (1): failing at the same point, with a mismatch in only one of the layers. Some major differences, though: