moby: Can't set net.ipv4.tcp_tw_reuse in docker 1.10.3

Output of docker version:

Client:
 Version:         1.10.3-el7.centos
 API version:     1.22
 Package version: docker-1.10.3-10.el7.centos.x86_64
 Go version:      go1.4.2
 Git commit:      0b4a971-unsupported
 Built:           Tue Jun 21 17:51:37 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3-el7.centos
 API version:     1.22
 Package version: docker-1.10.3-10.el7.centos.x86_64
 Go version:      go1.4.2
 Git commit:      0b4a971-unsupported
 Built:           Tue Jun 21 17:51:37 2016
 OS/Arch:         linux/amd64

Output of docker info:

[root@hh-yun-k8s-128225 ~]# docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 22
Server Version: 1.10.3-el7.centos
Storage Driver: devicemapper
 Pool Name: docker-8:2-135811130-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.691 GB
 Data Space Total: 107.4 GB
 Data Space Available: 74.34 GB
 Metadata Space Used: 7.533 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.14 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host netplugin overlay bridge
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 24
Total Memory: 188.7 GiB
Name: hh-yun-k8s-128225.vclound.com
ID: R7KV:QVNL:KZKZ:SJUS:ZGLR:FSXR:XEUB:AKOR:JB5G:WXRI:X7TW:T5V6
Cluster store: etcd://10.199.128.48:4001,10.199.128.49:4001,10.199.128.50:4001
Registries: docker.io (secure)

Additional environment details (AWS, VirtualBox, physical, etc.):

physical

Steps to reproduce the issue: 1.start container with --privilidged docker run --privilidged -it centos:6.6 /bin/bash 2.vim /etc/sysct.conf net.ipv4.tcp_tw_reuse=1 3.sysctl -p , got en error: net.ipv4.tcp_tw_reuse is an unknown key

Describe the results you received: net.ipv4.tcp_tw_reuse is an unknown key

Describe the results you expected: no error.

Additional information you deem important (e.g. issue happens only occasionally):

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

That quote is from a comment made before my quote, not after.

There are multiple examples of sysctls that don’t seem to be supported by docker at least on 4.14.114-103.97.amzn2.x86_64

[ec2-user@ip-10-91-3-193 ~]$ cat sysctls_clean.csv
net.ipv4.tcp_max_syn_backlog,100000
net.core.somaxconn,20000
net.core.netdev_max_backlog,100000
net.ipv4.tcp_slow_start_after_idle,0
net.ipv4.tcp_rmem,371967 495956 4194304
net.ipv4.tcp_wmem,371967 495956 4194304
net.ipv4.tcp_mem,4194304 4718592 5242880
net.ipv4.ip_local_port_range,10240 65535
net.ipv4.tcp_congestion_control,bbr
net.ipv4.tcp_low_latency,1
net.netfilter.nf_conntrack_max,2000000
net.netfilter.nf_conntrack_tcp_timeout_established,8000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait,10
net.netfilter.nf_conntrack_tcp_timeout_time_wait,10
net.netfilter.nf_conntrack_generic_timeout,60
net.ipv4.tcp_fin_timeout,10
net.ipv4.tcp_syn_retries,2
net.ipv4.tcp_synack_retries,2
net.ipv4.tcp_mtu_probing,1
net.ipv4.tcp_tw_reuse,1
net.ipv4.tcp_max_tw_buckets,262144
net.ipv4.tcp_abort_on_overflow,0
vm.min_free_kbytes,2000000

[ec2-user@ip-10-91-3-193 ~]$ while IFS=, read -r col1 col2; do docker run --sysctl "$col1=$col2" alpine:latest sysctl $col1; done < sysctls_clean.csv
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
e7c96db7181b: Pull complete
Digest: sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6
Status: Downloaded newer image for alpine:latest
net.ipv4.tcp_max_syn_backlog = 100000
net.core.somaxconn = 20000
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /proc/sys/net/core/netdev_max_backlog: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /proc/sys/net/ipv4/tcp_slow_start_after_idle: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled
net.ipv4.tcp_rmem = 371967	495956	4194304
net.ipv4.tcp_wmem = 371967	495956	4194304
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /proc/sys/net/ipv4/tcp_mem: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled
net.ipv4.ip_local_port_range = 10240	65535
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /proc/sys/net/ipv4/tcp_congestion_control: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /proc/sys/net/ipv4/tcp_low_latency: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled
net.netfilter.nf_conntrack_max = 2000000
net.netfilter.nf_conntrack_tcp_timeout_established = 8000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 10
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 10
net.netfilter.nf_conntrack_generic_timeout = 60
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 262144
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"open /proc/sys/net/ipv4/tcp_abort_on_overflow: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled
invalid argument "vm.min_free_kbytes=2000000" for "--sysctl" flag: sysctl 'vm.min_free_kbytes=2000000' is not whitelisted

Others seem to inherit their values from the host by default even though you can set them via --sysctl in docker, like:

[ec2-user@ip-10-91-3-193 ~]$ sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 371967	495956	4194304
[ec2-user@ip-10-91-3-193 ~]$ docker run alpine:latest sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 371967	495956	4194304

and the ones that you really need to set otherwise they will revert to the OS default (not the host value), like net.ipv4.tcp_tw_reuse:

[ec2-user@ip-10-91-3-193 ~]$ sysctl net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_reuse = 1
[ec2-user@ip-10-91-3-193 ~]$ docker run alpine:latest sysctl net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_reuse = 0

the whole situation is a mess and I would guess it already caused quite a few problems in multiple places.

Anyway, I just wanted to point out that, at least on some kernels, net.ipv4.tcp_tw_reuse is not inherited from the host, contrary to what was said here, so hopefully people who are looking for this info are not mislead by that.

All the net.* sysctls are namespaced, so the set available inside containers is not the same as on the host, depending if this is a global setting or not. This is a global setting which you need to change on the host.

Note for the ones that are namespaced, Docker 1.12 has a docker run --sysctl=... option to set them directly on container run, but this will not help you here.