distribution: Corrupt registry with Swift Storage

We have a few registries running on swift storage.

Some of the images can’t be pushed because of the error unknown error completing upload: swift: Timeout expired while waiting for segments of

c11090b4d522: Layer already exists 
0a6d02aaf2e3: Pushing [==================================================>] 74.66 MB
0a6d02aaf2e3: Pushing [===================================>               ] 52.33 MB/74.65 MB
da64abd44f85: Layer already exists 
11de431551d8: Layer already exists 
57dd1622cd6e: Layer already exists 
ac024582817f: Layer already exists 
67852b7d2cf5: Layer already exists 
f1f25d8c6e56: Layer already exists 
33a212e8aa56: Layer already exists 
eaaf72983326: Layer already exists 
a85f35566a26: Layer already exists

The layer 0a6d02aaf2e3 fails to upload with the error message.

Here is the full log with

registry_1    | time="2017-02-08T22:02:11Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=dockerhub.acme.com http.request.id=8d99fa24-64e6-4f89-ab16-98c9c5bffa03 http.request.method=PATCH http.request.remoteaddr=80.72.104.111 http.request.uri="/v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=Gsxmp39D3eeQZlYd8XBbaLPCMhjJuVAPpnpKIrqyBpV7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTAyLTA4VDIyOjAxOjM4LjM3MDc5Njk1OVoifQ%3D%3D" http.request.useragent="docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))" http.response.duration=8.407953144s http.response.status=202 http.response.written=0 instance.id=55615968-323e-432a-a720-bb36c0e127a7 version=v2.6.0 
registry_1    | 172.18.0.3 - - [08/Feb/2017:22:02:03 +0000] "PATCH /v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=Gsxmp39D3eeQZlYd8XBbaLPCMhjJuVAPpnpKIrqyBpV7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTAyLTA4VDIyOjAxOjM4LjM3MDc5Njk1OVoifQ%3D%3D HTTP/1.1" 202 0 "" "docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))"
proxy_1       | 80.72.104.111 - - [08/Feb/2017:22:02:12 +0000] "PATCH /v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=Gsxmp39D3eeQZlYd8XBbaLPCMhjJuVAPpnpKIrqyBpV7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTAyLTA4VDIyOjAxOjM4LjM3MDc5Njk1OVoifQ%3D%3D HTTP/1.1" 202 0 "-" "docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \x5C(darwin\x5C))"
registry_1    | time="2017-02-08T22:02:25Z" level=error msg="unknown error completing upload: swift: Timeout expired while waiting for segments of /docker/registry/v2/blobs/sha256/f2/f2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9/data to show up" auth.user.name=dev go.version=go1.7.3 http.request.host=dockerhub.acme.com http.request.id=798cb2f9-343d-4b2e-9eaf-98f949a4b187 http.request.method=PUT http.request.remoteaddr=80.72.104.111 http.request.uri="/v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=8xrLlv2lBSMuE5SJDiRwq9fHyqJHFOqpwPrxV9Rc17R7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0Ijo2Njc3NTU5NiwiU3RhcnRlZEF0IjoiMjAxNy0wMi0wOFQyMjowMTozOFoifQ%3D%3D&digest=sha256%3Af2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9" http.request.useragent="docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))" instance.id=55615968-323e-432a-a720-bb36c0e127a7 vars.name="acme/papyrus" vars.uuid=394d7fd6-5205-4fac-a45f-2561a4f31655 version=v2.6.0 
registry_1    | time="2017-02-08T22:02:25Z" level=error msg="error canceling upload after error: already closed" auth.user.name=dev go.version=go1.7.3 http.request.host=dockerhub.acme.com http.request.id=798cb2f9-343d-4b2e-9eaf-98f949a4b187 http.request.method=PUT http.request.remoteaddr=80.72.104.111 http.request.uri="/v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=8xrLlv2lBSMuE5SJDiRwq9fHyqJHFOqpwPrxV9Rc17R7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0Ijo2Njc3NTU5NiwiU3RhcnRlZEF0IjoiMjAxNy0wMi0wOFQyMjowMTozOFoifQ%3D%3D&digest=sha256%3Af2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9" http.request.useragent="docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))" instance.id=55615968-323e-432a-a720-bb36c0e127a7 vars.name="acme/papyrus" vars.uuid=394d7fd6-5205-4fac-a45f-2561a4f31655 version=v2.6.0 
registry_1    | time="2017-02-08T22:02:26Z" level=error msg="response completed with error" auth.user.name=dev err.code=unknown err.detail="swift: Timeout expired while waiting for segments of /docker/registry/v2/blobs/sha256/f2/f2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9/data to show up" err.message="unknown error" go.version=go1.7.3 http.request.host=dockerhub.acme.com http.request.id=798cb2f9-343d-4b2e-9eaf-98f949a4b187 http.request.method=PUT http.request.remoteaddr=80.72.104.111 http.request.uri="/v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=8xrLlv2lBSMuE5SJDiRwq9fHyqJHFOqpwPrxV9Rc17R7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0Ijo2Njc3NTU5NiwiU3RhcnRlZEF0IjoiMjAxNy0wMi0wOFQyMjowMTozOFoifQ%3D%3D&digest=sha256%3Af2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9" http.request.useragent="docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=13.641375316s http.response.status=500 http.response.written=104 instance.id=55615968-323e-432a-a720-bb36c0e127a7 vars.name="acme/papyrus" vars.uuid=394d7fd6-5205-4fac-a45f-2561a4f31655 version=v2.6.0 
registry_1    | 172.18.0.3 - - [08/Feb/2017:22:02:12 +0000] "PUT /v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=8xrLlv2lBSMuE5SJDiRwq9fHyqJHFOqpwPrxV9Rc17R7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0Ijo2Njc3NTU5NiwiU3RhcnRlZEF0IjoiMjAxNy0wMi0wOFQyMjowMTozOFoifQ%3D%3D&digest=sha256%3Af2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9 HTTP/1.1" 500 104 "" "docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))"
proxy_1       | 80.72.104.111 - - [08/Feb/2017:22:02:26 +0000] "PUT /v2/acme/papyrus/blobs/uploads/394d7fd6-5205-4fac-a45f-2561a4f31655?_state=8xrLlv2lBSMuE5SJDiRwq9fHyqJHFOqpwPrxV9Rc17R7Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiMzk0ZDdmZDYtNTIwNS00ZmFjLWE0NWYtMjU2MWE0ZjMxNjU1IiwiT2Zmc2V0Ijo2Njc3NTU5NiwiU3RhcnRlZEF0IjoiMjAxNy0wMi0wOFQyMjowMTozOFoifQ%3D%3D&digest=sha256%3Af2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9 HTTP/1.1" 500 104 "-" "docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \x5C(darwin\x5C))"
registry_1    | time="2017-02-08T22:02:31Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=dockerhub.acme.com http.request.id=5a42e5ed-99f2-4fee-be4e-57a5f002aafa http.request.method=POST http.request.remoteaddr=80.72.104.111 http.request.uri="/v2/acme/papyrus/blobs/uploads/" http.request.useragent="docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))" http.response.duration=457.531335ms http.response.status=202 http.response.written=0 instance.id=55615968-323e-432a-a720-bb36c0e127a7 version=v2.6.0 
proxy_1       | 80.72.104.111 - - [08/Feb/2017:22:02:31 +0000] "POST /v2/acme/papyrus/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \x5C(darwin\x5C))"
registry_1    | 172.18.0.3 - - [08/Feb/2017:22:02:31 +0000] "POST /v2/acme/papyrus/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/1.13.0 go/go1.7.3 git-commit/49bf474 kernel/4.9.4-moby os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.0 \\(darwin\\))"
proxy_1       | 2017/02/08 22:02:32 [warn] 27#27: *18743 a client request body is buffered to a temporary file /usr/local/openresty/nginx/client_body_temp/0000000154, client: 80.72.104.111, server: dockerhub.acme.com, request: "PATCH /v2/acme/papyrus/blobs/uploads/42564a13-49cd-4c9a-ae85-33f753e7de01?_state=d8xl8oIwSAjI-kXDlrSHQYtH7m37gncrcIGuBAPg0E97Ik5hbWUiOiJmaW5vL3BhcHlydXMiLCJVVUlEIjoiNDI1NjRhMTMtNDljZC00YzlhLWFlODUtMzNmNzUzZTdkZTAxIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTAyLTA4VDIyOjAyOjMxLjM1MDI1NjM3OFoifQ%3D%3D HTTP/1.1", host: "dockerhub.acme.com"

Investigating the failed blob

rootÉ2d6ca6783ab2:ü# swift stat --lh images  files/docker/registry/v2/blobs/sha256/f2/f2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9/data
       Account: AUTH_b50accfce5044d6e9e47f80d012ca4e7
     Container: images
        Object: files/docker/registry/v2/blobs/sha256/f2/f2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9/data
  Content Type: application/octet-stream
Content Length: 0
 Last Modified: Thu, 26 Jan 2017 10:36:46 GMT
          ETag: "d41d8cd98f00b204e9800998ecf8427e"
      Manifest: images/segments/2f6/46f636b65722f72656769737472792f76322f7265706f7369746f726965732f66696e6f2f706170797275732f5f75706c6f6164732f33363465623039332d663864642d346136352d613839302d6535633462396164333230632f646174616acec982c140da3457079e3682729fa43f5d2e8d81bd27ac4962d6f39f9f00f9da39a3ee5e6b4b0d3255bfef95601890afd80709
 Accept-Ranges: bytes
   X-Timestamp: 1485427005.63842
    X-Trans-Id: txc80646d37cbf4c31a8dba-00589baabc

After that I tried to download the Manifest: images/segments/2f6/46f63...

swift stat --lh images segments/2f6/46f636b6.... Object images/segments/2f6/46f636b657... not found

Should there be always a manifest?
Can I recreate the manifest or push it?
How can I find out where the layer f2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9 belongs to, because then I could delete the whole image.
How can I possibly fix the registry?

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 4
Comments: 24

Commits related to this issue

swift: extra consistency check after Move We face a consistency issue with our data store, time to time we end up with empty data file. This problem is similar to #2188. The upload of a blob is done... — committed to goneri/distribution by goneri 6 years ago

Most upvoted comments

Why did you close it ? Even if you solved the problem using another mean, the issue is still here. I am using Swift and exactly same issue with segment keep retrying…

Tronix117 on Aug 24, 2017

We are still suffering of this problem. If you need any details, please refer, i will provide information you need to fix this. We have intensively used registry setup with swift backend and unable to clean it properly

shcherbak on May 12, 2022

I have the same issue with swift storage,however,It can be fixed by two ways:

delete all the wrong layers ,we use the failed layers above for example:

0a6d02aaf2e3: Pushing [==================================================>] 74.66 MB
0a6d02aaf2e3: Pushing [===================================>               ] 52.33 MB/74.65 MB

First , get the digest from diffid 0a6d02aaf2e3 in ${docker root dir }/images/xxx/distribution/v2metadata-by-diffid/sha256/0a/0a6d02aaf2e3xxxxxxxxx

Second,use the swift command to delete the corrupt object

swift delete ${container_name} path_of_object
swfit delete images files/docker/registry/v2/blobs/sha256/f2/f2327c65e7abe07a0f63af4c87750e1e7d99eac14344588b8df03ff716fd5ea9/data

Third, repush the image . if get an http 500 error at last ,you shoud also delete the image id object data

   docker inspect image to get imageid at top "Id":"sha256:xxxxxxx"
  swift delete images files/docker/registry/v2/blobs/sha256/xx/xxxxxx.

and repush image again

Also ,you can modify Stat function in swift.go. like below,

if isDLO && info.Bytes == 0 {
			if time.Now().Add(waitingTime).After(endTime) {
				fmt.Printf("Timeout expired while waiting for segments of %s to show up,Just delete it and upload it again\n", path)
				err = d.Delete(ctx, path)
				if err == nil {
					fmt.Printf("delete %s done!!!!\n", path)
					return nil, storagedriver.PathNotFoundError{Path: path}
				}
				fmt.Printf("delete %s error,the err is %s\n", path, err)
				return nil, fmt.Errorf("Timeout expired while waiting for segments of %s to show up", path)
			}
			time.Sleep(waitingTime)
			waitingTime *= 2
			continue
		}

The two ways works for me, Hope it will be helpful to you.

zhxqgithub on May 9, 2019