moby: Pulling image from V1 registry with Docker 1.10 and then pushing to a V2 registry (or running docker save) sometimes results in error: "could not verify layer data"
Output of docker version:
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.1
Git commit: 20f81dd
Built: Thu Jun 30 21:32:57 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.1
Git commit: 20f81dd
Built: Thu Jun 30 21:32:57 2016
OS/Arch: linux/amd64
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 39
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: docker-252:2-6817609-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 9.188 GB
Data Space Total: 107.4 GB
Data Space Available: 98.19 GB
Metadata Space Used: 15.41 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.132 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.117-RHEL6 (2016-08-15)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.18.2.el7.x86_64
Operating System: <unknown>
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 19.45 GiB
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Additional environment details (AWS, VirtualBox, physical, etc.):
Steps to reproduce the issue:
- Pull image from a V1 Docker registry
- Tag and push the image to a V2 Docker registry with Content Trust enabled
Describe the results you received: Push fails with an error such as the following in /var/log/docker:
Upload failed: could not verify layer data for: sha256:a566d6467316a731f621511e8b3095449404392c6a3260695bea1f37b6e24a63. This may be because internal files in the layer store were modified. Re-pulling or rebuilding this image may resolve the issue
Describe the results you expected: Successful push.
Additional information you deem important (e.g. issue happens only occasionally): Push fails only on certain image layers.
Push succeeds also fails if Content Trust is not enabled.
Images relying on the problematic layers appear to function fine when run in a container.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 34 (19 by maintainers)
@stevvooe No, this happens on layerstore TarStream level. It verifies that bytes coming out of layerstore are the same that were used in
layerstore.Register. Compression and talking with registry is outside layerstore. This can only be 1) bug in tar-split 2) bug in layerstore 3) external modification of data on disk.@TanmayaA: This has been tracked down to a bug in the
tar-splitlibrary. We’re working on a fix fortar-splitand should submit it upstream soon.Thanks again for providing an image that exhibited the problem.
@aaronlehmann Since this docker build only has to run on a docker migrator, I think I’m good using the nightly build. Will be great if you can update here, along with usage instructions, when this fix is available there.