moby: Pulling image from V1 registry with Docker 1.10 and then pushing to a V2 registry (or running docker save) sometimes results in error: "could not verify layer data"
Output of docker version
:
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.1
Git commit: 20f81dd
Built: Thu Jun 30 21:32:57 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.1
Git commit: 20f81dd
Built: Thu Jun 30 21:32:57 2016
OS/Arch: linux/amd64
Output of docker info
:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 39
Server Version: 1.10.3
Storage Driver: devicemapper
Pool Name: docker-252:2-6817609-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 9.188 GB
Data Space Total: 107.4 GB
Data Space Available: 98.19 GB
Metadata Space Used: 15.41 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.132 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.117-RHEL6 (2016-08-15)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-327.18.2.el7.x86_64
Operating System: <unknown>
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 19.45 GiB
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Additional environment details (AWS, VirtualBox, physical, etc.):
Steps to reproduce the issue:
- Pull image from a V1 Docker registry
- Tag and push the image to a V2 Docker registry with Content Trust enabled
Describe the results you received: Push fails with an error such as the following in /var/log/docker:
Upload failed: could not verify layer data for: sha256:a566d6467316a731f621511e8b3095449404392c6a3260695bea1f37b6e24a63. This may be because internal files in the layer store were modified. Re-pulling or rebuilding this image may resolve the issue
Describe the results you expected: Successful push.
Additional information you deem important (e.g. issue happens only occasionally): Push fails only on certain image layers.
Push succeeds also fails if Content Trust is not enabled.
Images relying on the problematic layers appear to function fine when run in a container.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 34 (19 by maintainers)
@stevvooe No, this happens on layerstore TarStream level. It verifies that bytes coming out of layerstore are the same that were used in
layerstore.Register
. Compression and talking with registry is outside layerstore. This can only be 1) bug in tar-split 2) bug in layerstore 3) external modification of data on disk.@TanmayaA: This has been tracked down to a bug in the
tar-split
library. We’re working on a fix fortar-split
and should submit it upstream soon.Thanks again for providing an image that exhibited the problem.
@aaronlehmann Since this docker build only has to run on a docker migrator, I think I’m good using the nightly build. Will be great if you can update here, along with usage instructions, when this fix is available there.