distribution: High memory usage with s3 driver

The s3 driver seems to have a rather high memory usage when pushing and a few minutes after.

Tested with: docker pull golang:1.6 && docker tag .. && docker push With the default chunk size of 10MB peak memory is around 148MB, with a chunk size of 5MB peak memory is around 98MB.

Registry was started like:

docker run -t -i 
    -e REGISTRY_STORAGE=s3
    -e REGISTRY_STORAGE_S3_REGION=eu-central-1
    -e REGISTRY_STORAGE_S3_BUCKET=foo
    -e REGISTRY_STORAGE_S3_ACCESSKEY="foo"
    -e REGISTRY_STORAGE_S3_SECRETKEY="barr"
    -e REGISTRY_STORAGE_S3_V4AUTH=true
    -e REGISTRY_STORAGE_S3_CHUNKSIZE=5242880
    -e REGISTRY_LOG_LEVEL=debug
    -e GOGC=5
    --rm
    -p 5000:5000
    --name registry
    distribution/registry:master

pprof:

http://sprunge.us/fIZM (before)
http://sprunge.us/aNDg (under)
http://sprunge.us/giQI (under)
http://sprunge.us/GVII (under)
http://sprunge.us/XHdK (after with debug=2)
http://sprunge.us/cULA (after with debug=2)
http://sprunge.us/NHMh (after with debug=2, memory dropped to 21MB)

docker version/info and registry version

[kristian@arch-hp-laptop ~]$ docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Sat Mar 12 19:18:57 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Sat Mar 12 19:18:57 2016
 OS/Arch:      linux/amd64
[kristian@arch-hp-laptop ~]$ docker info
Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 126
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-8:3-6949349-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 5.265 GB
 Data Space Total: 107.4 GB
 Data Space Available: 102.1 GB
 Metadata Space Used: 9.867 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.138 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.119 (2016-03-04)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.5-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.32 GiB
Name: arch-hp-laptop
ID: MDNG:D2CL:76WB:JMFF:OESH:A7HR:LFWB:ISOI:R7XY:QDI6:3IHV:KHOS
Username: klausenbusk
Registry: https://index.docker.io/v1/
root@b237c8b87c8b:/go/src/github.com/docker/distribution# registry --version
registry github.com/docker/distribution 2c635d1