distribution: Docker pull i/o timeout with v2 registry

I’m running a private registry (v2) with the following configuration.

version: 0.1
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /tmp/registry-dev
    maintenance:
        uploadpurging:
            enabled: false
http:
    addr: :5000
    debug:
        addr: localhost:5001
auth:
  token:
    realm: https://thinknode.dev/v2/token/
    service: registry.thinknode.dev
    issuer: thinknode.dev
    rootcertbundle: /usr/local/share/ca-certificates/thinknode.dev/thinknode.dev.crt
redis:
  addr: localhost:6379
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms

The registry sits behind nginx which is configured as follows:

upstream docker {
  server localhost:5000;
}

server {
  listen       443;
  server_name  registry.thinknode.dev;

  add_header Docker-Distribution-API-Version registry/2.0 always;

  proxy_set_header Host       $http_host;    # Required for Docker client
  proxy_set_header X-Real-IP  $remote_addr;  # Passthrough real client IP

  client_max_body_size 0;        # Disable any limits to avoid HTTP 413 for large image uploads
  chunked_transfer_encoding on;  # Required to avoid HTTP 411

  ssl on;
  ssl_certificate /usr/local/share/ca-certificates/registry.thinknode.dev/registry.thinknode.dev.crt;
  ssl_certificate_key /etc/ssl/private/registry.thinknode.dev.key;

  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Original-URI $request_uri;
  proxy_set_header Docker-Distribution-Api-Version registry/2.0;

  location / {
    proxy_pass http://docker;
  }
}

Occasionally (about 15% of the time), pulling an image with the npm module dockerode fails with the following logs from the docker daemon.

ERRO[10608] Error from V2 registry: Get https://registry.thinknode.dev/v2/035088a9398b/math/manifests/sha256:10af39c70b7b7f3bc6d9539c38180d67a626c033487e706e5396d3c5af6c58a8: read tcp 127.0.0.1:443: i/o timeout 
ERRO[10608] Handler for POST /images/create returned error: pulling with digest reference failed from v2 registry 
ERRO[10608] HTTP Error                                    err=pulling with digest reference failed from v2 registry statusCode=500

Here is my docker version information:

Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64

Not sure where to go from here. Any suggestions?

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 20 (4 by maintainers)

Most upvoted comments

There was a bug, it was fixed and I replied in the other thread which @depay raised this issue in.