moby: Pulling image from V1 registry with Docker 1.10 and then pushing to a V2 registry (or running docker save) sometimes results in error: "could not verify layer data"

Output of docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.1
 Git commit:   20f81dd
 Built:        Thu Jun 30 21:32:57 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.1
 Git commit:   20f81dd
 Built:        Thu Jun 30 21:32:57 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 39
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-252:2-6817609-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 9.188 GB
 Data Space Total: 107.4 GB
Data Space Available: 98.19 GB
 Metadata Space Used: 15.41 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.132 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.117-RHEL6 (2016-08-15)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 3.10.0-327.18.2.el7.x86_64
Operating System: <unknown>
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 19.45 GiB
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

Steps to reproduce the issue:

  1. Pull image from a V1 Docker registry
  2. Tag and push the image to a V2 Docker registry with Content Trust enabled

Describe the results you received: Push fails with an error such as the following in /var/log/docker:

Upload failed: could not verify layer data for: sha256:a566d6467316a731f621511e8b3095449404392c6a3260695bea1f37b6e24a63. This may be because internal files in the layer store were modified. Re-pulling or rebuilding this image may resolve the issue

Describe the results you expected: Successful push.

Additional information you deem important (e.g. issue happens only occasionally): Push fails only on certain image layers.

Push succeeds also fails if Content Trust is not enabled.

Images relying on the problematic layers appear to function fine when run in a container.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 34 (19 by maintainers)

Commits related to this issue

Most upvoted comments

@stevvooe No, this happens on layerstore TarStream level. It verifies that bytes coming out of layerstore are the same that were used in layerstore.Register. Compression and talking with registry is outside layerstore. This can only be 1) bug in tar-split 2) bug in layerstore 3) external modification of data on disk.

@TanmayaA: This has been tracked down to a bug in the tar-split library. We’re working on a fix for tar-split and should submit it upstream soon.

Thanks again for providing an image that exhibited the problem.

@aaronlehmann Since this docker build only has to run on a docker migrator, I think I’m good using the nightly build. Will be great if you can update here, along with usage instructions, when this fix is available there.