moby: Error processing tar file(exit status 1): unexpected EOF adding an 8G file (8G - 1 byte is ok)

Description

This was first reported on Mac by @flagbug: https://github.com/docker/for-mac/issues/2388#issuecomment-409532891 but I believe it is a generic problem with the engine, not something specific to Mac.

To reproduce:

/tmp/test # cat Dockerfile
FROM ubuntu:18.04

RUN head -c 8G </dev/urandom >file

/tmp/test # docker build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:18.04
 ---> 735f80812f90
Step 2/2 : RUN head -c 8G </dev/urandom >file
 ---> Running in 4debf740eb97
Error processing tar file(exit status 1): unexpected EOF

Describe the results you received:

The error “Error processing tar file(exit status 1): unexpected EOF”

Describe the results you expected:

The build should complete successfully.

Additional information you deem important (e.g. issue happens only occasionally):

The logs have:

2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.598573889Z" level=debug msg="Tar with options on /var/lib/docker/overlay2/f89859e8af0b77222f209807333da05295ee03d1a9e2255895cf10ec57cce417/diff" storage-driver=overlay2
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.602482132Z" level=debug msg="Applying tar in /var/lib/docker/overlay2/278e40f6fc45ea9828cb090bf339b4f85561fa03dd2446626c3d9b6a272d8f5b/diff" storage-driver=overlay2
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.750205351Z" level=debug msg="Cleaning up layer 278e40f6fc45ea9828cb090bf339b4f85561fa03dd2446626c3d9b6a272d8f5b: Error processing tar file(exit status 1): unexpected EOF"
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.751542982Z" level=error msg="Can't add file /var/lib/docker/overlay2/f89859e8af0b77222f209807333da05295ee03d1a9e2255895cf10ec57cce417/diff/file to tar: io: read/write on closed pipe"
2018-08-02T14:16:35Z docker-ce time="2018-08-02T14:16:35.751599637Z" level=error msg="Can't close tar writer: io: read/write on closed pipe"

If I switch the command to head -c 8589934591 </dev/urandom >file i.e. 8GiB - 1 byte then it succeeds correctly.

I suspect this is same underlying issue as https://github.com/moby/moby/issues/37305 where a microsoft/mssql-server-linux database backup is restored and then docker committed. I suspect the root cause is that an individual file exceeds 8GiB, which is a limit in the tar header. According to wikipedia:

Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, star in 2001 introduced a base-256 coding that is indicated by setting the high-order bit of the leftmost byte of a numeric field. GNU-tar and BSD-tar followed this idea. Additionally, versions of tar from before the first POSIX standard from 1988 pad the values with spaces instead of zeroes.

Output of docker version:

/tmp/test # docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:04:40 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:13:46 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker info:

/tmp/test # docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.93-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: UFJG:WKJW:OBVN:JP4C:NXIX:YJ7Z:JHQS:AT4L:RO2C:MN7I:YMUW:EQN7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 34
 Goroutines: 68
 System Time: 2018-08-02T14:49:58.963009758Z
 EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):