distribution: registry:2 corrupts uploaded layers (filesystem layer verification failed) when using 3rd-party S3

Edit 2: It looks like an issue of handling multipart offset on minio’s part. The developers are actively trying to resolve the issue, and they proposed a fix (pending validation).

---

Edit 1: It is possible that 63468ef4a85fabb756e799c2534f0df2f3c8167c introduces this regression

---

Numerous similar issues: #2168, docker/docker#21290, etc

Running on Kubernetes cluster:

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+coreos.0", GitCommit:"cc65f5321f9230bf9a3fa171155c1213d6e3480e", GitTreeState:"clean", BuildDate:"2016-12-14T04:08:28Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Registry deployment file:

{
    "apiVersion": "extensions/v1beta1",
    "kind": "Deployment",
    "metadata": {
        "name": "kube-registry",
        "namespace": "kube-system"
    },
    "spec": {
        "replicas": 3,
        "selector": {
            "matchLabels": {
                "app": "kube-registry"
            }
        },
        "template": {
            "metadata": {
                "labels": {
                    "app": "kube-registry"
                }
            },
            "spec": {
                "volumes": [{
                    "name": "auth-signing",
                    "secret": {
                        "secretName": "auth-signing"
                    }
                }],
                "containers": [{
                    "name": "sysctl-buddy",
                    "image": "alpine:latest",
                    "command": [
                        "/bin/sh",
                        "-c",
                        "while true; do\n  sysctl -w net.core.somaxconn=32768 > /dev/null 2>&1\n  sleep 10\ndone\n"
                    ],
                    "imagePullPolicy": "IfNotPresent",
                    "securityContext": {
                        "privileged": true
                    }
                }, {
                    "name": "registry",
                    "image": "registry:2",
                    "resources": {
                        "requests": {
                            "cpu": "200m",
                            "memory": "1024Mi"
                        },
                        "limits": {
                            "cpu": "2",
                            "memory": "2048Mi"
                        }
                    },
                    "env": [{
                            "name": "REGISTRY_LOG_LEVEL",
                            "value": "warn"
                        },
                        {
                            "name": "REGISTRY_HTTP_ADDR",
                            "value": ":5000"
                        },
                        {
                            "name": "REGISTRY_HTTP_HOST",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_HTTP_SECRET",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_STORAGE",
                            "value": "s3"
                        },
                        {
                            "name": "REGISTRY_STORAGE_DELETE_ENABLED",
                            "value": "true"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_REGION",
                            "value": "us-east-1"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_REGIONENDPOINT",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_BUCKET",
                            "value": "images"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_ACCESSKEY",
                            "value": "docker"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_SECRETKEY",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_ENCRYPT",
                            "value": "false"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_SECURE",
                            "value": "false"
                        },
                        {
                            "name": "REGISTRY_STORAGE_S3_CHUNKSIZE",
                            "value": "20971520"
                        },
                        {
                            "name": "REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR",
                            "value": "redis"
                        },
                        {
                            "name": "REGISTRY_REDIS_ADDR",
                            "value": "docker-redis-cache:6379"
                        },
                        {
                            "name": "REGISTRY_AUTH",
                            "value": "token"
                        },
                        {
                            "name": "REGISTRY_AUTH_TOKEN_REALM",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_AUTH_TOKEN_SERVICE",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_AUTH_TOKEN_ISSUER",
                            "value": "[redacted]"
                        },
                        {
                            "name": "REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE",
                            "value": "/auth/auth.crt"
                        }
                    ],
                    "ports": [{
                        "containerPort": 5000,
                        "name": "registry",
                        "protocol": "TCP"
                    }],
                    "volumeMounts": [{
                        "name": "auth-signing",
                        "mountPath": "/auth",
                        "readOnly": true
                    }]
                }]
            }
        }
    }
}

Registry running with a 3rd-party S3 backend (minio).

---

Findings: registry:2 breaks any layers larger than 100MB (docker push is fine, however docker pull gives error).

However, changing the image to registry:2.5.1 or registry:2.5, and no more corruptions happened. Something must have changed between 2.5.x and 2.6.