distribution: Registry 2.6.2 with s3 backend panic: runtime error: invalid memory address or nil pointer dereference

Had the same issue (using the helm package) with registry:v2.7.1

For now I’ve forced the 2.5.2 image which seems to work:

 helm upgrade --install --set image.tag=2.5.2 docker-registry stable/docker-registry -f chart_values.yaml

Any news on when / which version this will be fixed? Latest tag today is still 2.7.1 (3 months old)

I’m also running into issues with registry panic’ing when talking to s3, and it works in master. What’s keeping from cutting a new release?

Any news on this?

I switched over to https://gitlab.com/gitlab-org/container-registry/

2.7.1 is more like 9 months old. Very sad that there is so little movement in the various issues

Can you please spin a new release with the updated S3 SDK ?

Had this problem when using DigitalOcean Spaces, which appears to be Ceph under the covers. 2.5.2 works, though.

Encountered this issue recently with Ceph’s RADOSGW S3 storage. 2.6.2 works just fine, but 2.7 encounters the panic.

[edit]

To expand, this is using Ceph 12.2.12 without the use of the civetvweb frontend.

@oanogin 2.5.2 works fine

@ederst GitLab PM for the registry here. Actually, after surveying GitLab users, we’ve decided not to deprecate the proxy feature. Hopefully, that helps you stick with the GitLab Container Registry. Ping me if you have any issues.

I useregistry:2.7.1 with DO Spaces S3 backend. I see the same error when requesting GET /v2/_catalog. Otherwise everything seems to be working fine. UPD. /bin/registry garbage-collect /etc/docker/registry/config.yml fails too

The Ceph issue @r0ss3 references is supposedly fixed in Ceph 13.2.5 but I am still seeing this issue with that version of Ceph.

Same stroy as with @pilosus Using 2.7.1 with DO S3 getting error on /v2/_catalog

Downgrading to registry:2.5.2 solved the issue!

@pilosus , you should try.

P.S. Stacktrace:

time="2019-07-05T00:37:45.269923822Z" level=info msg="authorized request" go.version=go1.11.2 http.request.host="XXXXXXX:5000" http.request.id=97f5355c-73a6-46a8-a8a8-ead6b2e0d4b7 http.request.method=GET http.request.remoteaddr="10.255.0.4:58994" http.request.uri="/v2/_catalog" http.request.useragent="PostmanRuntime/7.15.0" 

time="2019-07-05T00:37:45.343278601Z" level=panic msg="runtime error: invalid memory address or nil pointer dereference" 

2019/07/05 00:37:45 http: panic serving 10.255.0.4:58994: &{0xc000088230 map[] 2019-07-05 00:37:45.343278601 +0000 UTC m=+962.287454739 panic runtime error: invalid memory address or nil pointer dereference <nil>}

goroutine 49 [running]:
net/http.(*conn).serve.func1(0xc0002588c0)
	/usr/local/go/src/net/http/server.go:1746 +0xd0
panic(0xe8e920, 0xc000089b80)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/docker/distribution/vendor/github.com/sirupsen/logrus.Entry.log(0xc000088230, 0xc0007483c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/docker/distribution/vendor/github.com/sirupsen/logrus/entry.go:124 +0x574
github.com/docker/distribution/vendor/github.com/sirupsen/logrus.(*Entry).Panic(0xc0000880f0, 0xc0004e6b88, 0x1, 0x1)
	/go/src/github.com/docker/distribution/vendor/github.com/sirupsen/logrus/entry.go:169 +0xb2
github.com/docker/distribution/vendor/github.com/sirupsen/logrus.(*Logger).Panic(0xc000088230, 0xc0004e6b88, 0x1, 0x1)
	/go/src/github.com/docker/distribution/vendor/github.com/sirupsen/logrus/logger.go:236 +0x6d
github.com/docker/distribution/vendor/github.com/sirupsen/logrus.Panic(0xc0004e6b88, 0x1, 0x1)
	/go/src/github.com/docker/distribution/vendor/github.com/sirupsen/logrus/exported.go:107 +0x4b
github.com/docker/distribution/registry.panicHandler.func1.1()
	/go/src/github.com/docker/distribution/registry/registry.go:345 +0xf9
panic(0xd81c00, 0x170a620)
	/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk.func1(0xc0000a6d80, 0xc000285301, 0xc0005f4a80)
	/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:973 +0x9d
github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3.(*S3).ListObjectsV2PagesWithContext(0xc00000cb48, 0x7f4359e4ef48, 0xc000281b20, 0xc000088af0, 0xc0004e7178, 0x0, 0x0, 0x0, 0x1, 0x2)
	/go/src/github.com/docker/distribution/vendor/github.com/aws/aws-sdk-go/service/s3/api.go:4198 +0x111
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).doWalk(0xc000446980, 0xff1ae0, 0xc000281ab0, 0xc0004e7238, 0xc000244691, 0x20, 0xea6655, 0x1, 0xc000088aa0, 0x0, ...)
	/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:971 +0x3a1
github.com/docker/distribution/registry/storage/driver/s3-aws.(*driver).Walk(0xc000446980, 0xff1ae0, 0xc000281ab0, 0xc0005e8e20, 0x20, 0xc000088aa0, 0x2, 0x0)
	/go/src/github.com/docker/distribution/registry/storage/driver/s3-aws/s3.go:919 +0x160
github.com/docker/distribution/registry/storage/driver/base.(*Base).Walk(0xc0003edb50, 0xff1ae0, 0xc000281ab0, 0xc0005e8e20, 0x20, 0xc000088aa0, 0x0, 0x0)
	/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go:239 +0x234
github.com/docker/distribution/registry/storage.(*registry).Repositories(0xc00046a150, 0xff1c60, 0xc00031c960, 0xc000072a00, 0x64, 0x64, 0x0, 0x0, 0x10015c0, 0x1, ...)
	/go/src/github.com/docker/distribution/registry/storage/catalog.go:30 +0x1fb
github.com/docker/distribution/registry/handlers.(*catalogHandler).GetCatalog(0xc00000c138, 0xfef7a0, 0xc000284240, 0xc000319000)
	/go/src/github.com/docker/distribution/registry/handlers/catalog.go:48 +0x12a
github.com/docker/distribution/registry/handlers.(*catalogHandler).GetCatalog-fm(0xfef7a0, 0xc000284240, 0xc000319000)
	/go/src/github.com/docker/distribution/registry/handlers/catalog.go:24 +0x48
net/http.HandlerFunc.ServeHTTP(0xc0005f4a00, 0xfef7a0, 0xc000284240, 0xc000319000)
	/usr/local/go/src/net/http/server.go:1964 +0x44
github.com/docker/distribution/vendor/github.com/gorilla/handlers.MethodHandler.ServeHTTP(0xc000749200, 0xfef7a0, 0xc000284240, 0xc000319000)
	/go/src/github.com/docker/distribution/vendor/github.com/gorilla/handlers/handlers.go:35 +0x342
github.com/docker/distribution/registry/handlers.(*App).dispatcher.func1(0xfef7a0, 0xc000284240, 0xc000318700)
	/go/src/github.com/docker/distribution/registry/handlers/app.go:726 +0x51f
net/http.HandlerFunc.ServeHTTP(0xc000430560, 0xfef7a0, 0xc000284240, 0xc000318700)
	/usr/local/go/src/net/http/server.go:1964 +0x44
github.com/docker/distribution/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc000267090, 0xfef7a0, 0xc000284240, 0xc000318700)
	/go/src/github.com/docker/distribution/vendor/github.com/gorilla/mux/mux.go:114 +0xe0
github.com/docker/distribution/registry/handlers.(*App).ServeHTTP(0xc0003db180, 0xff0b60, 0xc000600000, 0xc000318300)
	/go/src/github.com/docker/distribution/registry/handlers/app.go:640 +0x297
github.com/docker/distribution/registry.alive.func1(0xff0b60, 0xc000600000, 0xc000318300)
	/go/src/github.com/docker/distribution/registry/registry.go:365 +0x6a
net/http.HandlerFunc.ServeHTTP(0xc0003e7110, 0xff0b60, 0xc000600000, 0xc000318300)
	/usr/local/go/src/net/http/server.go:1964 +0x44
github.com/docker/distribution/health.Handler.func1(0xff0b60, 0xc000600000, 0xc000318300)
	/go/src/github.com/docker/distribution/health/health.go:271 +0x11a
net/http.HandlerFunc.ServeHTTP(0xc000430f20, 0xff0b60, 0xc000600000, 0xc000318300)
	/usr/local/go/src/net/http/server.go:1964 +0x44
github.com/docker/distribution/registry.panicHandler.func1(0xff0b60, 0xc000600000, 0xc000318300)
	/go/src/github.com/docker/distribution/registry/registry.go:348 +0x81
net/http.HandlerFunc.ServeHTTP(0xc000430f40, 0xff0b60, 0xc000600000, 0xc000318300)
	/usr/local/go/src/net/http/server.go:1964 +0x44
net/http.serverHandler.ServeHTTP(0xc0003a3d40, 0xff0b60, 0xc000600000, 0xc000318300)
	/usr/local/go/src/net/http/server.go:2741 +0xab
net/http.(*conn).serve(0xc0002588c0, 0xff1760, 0xc000284080)
	/usr/local/go/src/net/http/server.go:1847 +0x646
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x2f5