calico: MTU for Calico Interfaces in Calico 3.2.1 not being recognized

Expected Behavior

When updating the veth_mtu to 8981 on AWS with jumbo frames enabled expect the calico interface. This worked previously in version Calico 3.1

Current Behavior

After updating every mtu value in the daemonset and configmap new calico interfaces are brought up with an mtu of 1500.

Showing interfaces brought up and down default to 1500

root@ip-172-31-0-160:/etc/cni/net.d# date
Thu Sep 20 16:42:20 UTC 2018
root@ip-172-31-0-160:/etc/cni/net.d# ifconfig | grep -A 2 cali
cali0ab4c0cd5c4 Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
          inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:8981  Metric:1
--
cali63b7f4dbf1a Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
          inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:8981  Metric:1
root@ip-172-31-0-160:/etc/cni/net.d# date
Thu Sep 20 16:42:47 UTC 2018
root@ip-172-31-0-160:/etc/cni/net.d# ifconfig | grep -A 2 cali
cali0ab4c0cd5c4 Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
          inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:8981  Metric:1
--
cali0dc4dd70749 Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
          inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
--
cali63b7f4dbf1a Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
          inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:8981  Metric:1
--
calia1f7ebc010a Link encap:Ethernet  HWaddr ee:ee:ee:ee:ee:ee
          inet6 addr: fe80::ecee:eeff:feee:eeee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          
root@ip-172-31-0-160:/etc/cni/net.d# cat 10-calico.conflist
{
  "name": "k8s-pod-network",
  "cniVersion": "0.3.0",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "datastore_type": "kubernetes",
      "nodename": "ip-172-31-0-160",
      "mtu": "8981",
      "ipam": {
        "type": "host-local",
        "subnet": "usePodCidr"
      },
      "policy": {
          "type": "k8s"
      },
      "kubernetes": {
          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      }
    },
    {
      "type": "portmap",
      "snat": true,
      "capabilities": {"portMappings": true}
    }
  ]
}

Calico-config map

apiVersion: v1
data:
calico_backend: bird
cni_network_config: |-
{
"name": "k8s-pod-network",
"cniVersion": "0.3.0",
"plugins": [
{
"type": "calico",
"log_level": "info",
"datastore_type": "kubernetes",
"nodename": "KUBERNETES_NODE_NAME",
"mtu": "8981",
"ipam": {
"type": "host-local",
"subnet": "usePodCidr"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "KUBECONFIG_FILEPATH"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {"portMappings": true}
}
]
}
typha_service_name: none
veth_mtu: "8981"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"calico_backend":"bird","cni_network_config":"{\n "name": "k8s-pod-network",\n "cniVersion": "0.3.0",\n "plugins": [\n {\n "type": "calico",\n "log_level": "info",\n "datastore_type": "kubernetes",\n "nodename": "KUBERNETES_NODE_NAME",\n "mtu": "8981",\n "ipam": {\n "type": "host-local",\n "subnet": "usePodCidr"\n },\n "policy": {\n "type": "k8s"\n },\n "kubernetes": {\n "kubeconfig": "KUBECONFIG_FILEPATH"\n }\n },\n {\n "type": "portmap",\n "snat": true,\n "capabilities": {"portMappings": true}\n }\n ]\n}","typha_service_name":"none","veth_mtu":"8981"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"calico-config","namespace":"kube-system"}}
creationTimestamp: 2018-09-20T15:27:51Z
name: calico-config
namespace: kube-system
resourceVersion: "319135"
selfLink: /api/v1/namespaces/kube-system/configmaps/calico-config
uid: bc87a3c1-bce9-11e8-ac07-0210da4b9e32

Calico Daemonset

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"DaemonSet","metadata":{"annotations":{},"labels":{"k8s-app":"calico-node"},"name":"calico-node","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"calico-node"}},"template":{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/critical-pod":""},"labels":{"k8s-app":"calico-node"}},"spec":{"containers":[{"env":[{"name":"DATASTORE_TYPE","value":"kubernetes"},{"name":"FELIX_TYPHAK8SSERVICENAME","valueFrom":{"configMapKeyRef":{"key":"typha_service_name","name":"calico-config"}}},{"name":"WAIT_FOR_DATASTORE","value":"true"},{"name":"NODENAME","valueFrom":{"fieldRef":{"fieldPath":"spec.nodeName"}}},{"name":"CALICO_NETWORKING_BACKEND","valueFrom":{"configMapKeyRef":{"key":"calico_backend","name":"calico-config"}}},{"name":"CLUSTER_TYPE","value":"k8s,bgp"},{"name":"IP","value":"autodetect"},{"name":"CALICO_IPV4POOL_IPIP","value":"Always"},{"name":"FELIX_IPINIPENABLED","value":"true"},{"name":"FELIX_IPINIPMTU","value":"8981"},{"name":"CALICO_IPV4POOL_CIDR","value":"10.244.0.0/16"},{"name":"CALICO_DISABLE_FILE_LOGGING","value":"true"},{"name":"FELIX_DEFAULTENDPOINTTOHOSTACTION","value":"ACCEPT"},{"name":"FELIX_IPV6SUPPORT","value":"false"},{"name":"FELIX_LOGSEVERITYSCREEN","value":"info"},{"name":"FELIX_HEALTHENABLED","value":"true"}],"image":"quay.io/calico/node:v3.2.1","livenessProbe":{"failureThreshold":6,"httpGet":{"host":"localhost","path":"/liveness","port":9099},"initialDelaySeconds":10,"periodSeconds":10},"name":"calico-node","readinessProbe":{"exec":{"command":["/bin/calico-node","-bird-ready","-felix-ready"]},"periodSeconds":10},"resources":{"requests":{"cpu":"250m"}},"securityContext":{"privileged":true},"volumeMounts":[{"mountPath":"/lib/modules","name":"lib-modules","readOnly":true},{"mountPath":"/var/run/calico","name":"var-run-calico","readOnly":false},{"mountPath":"/var/lib/calico","name":"var-lib-calico","readOnly":false}]},{"command":["/install-cni.sh"],"env":[{"name":"CNI_CONF_NAME","value":"10-calico.conflist"},{"name":"KUBERNETES_NODE_NAME","valueFrom":{"fieldRef":{"fieldPath":"spec.nodeName"}}},{"name":"CNI_NETWORK_CONFIG","valueFrom":{"configMapKeyRef":{"key":"cni_network_config","name":"calico-config"}}},{"name":"CNI_MTU","valueFrom":{"configMapKeyRef":{"key":"veth_mtu","name":"calico-config"}}}],"image":"quay.io/calico/cni:v3.2.1","name":"install-cni","volumeMounts":[{"mountPath":"/host/opt/cni/bin","name":"cni-bin-dir"},{"mountPath":"/host/etc/cni/net.d","name":"cni-net-dir"}]}],"hostNetwork":true,"nodeSelector":{"beta.kubernetes.io/os":"linux"},"serviceAccountName":"calico-node","terminationGracePeriodSeconds":0,"tolerations":[{"effect":"NoSchedule","operator":"Exists"},{"key":"CriticalAddonsOnly","operator":"Exists"},{"effect":"NoExecute","operator":"Exists"}],"volumes":[{"hostPath":{"path":"/lib/modules"},"name":"lib-modules"},{"hostPath":{"path":"/var/run/calico"},"name":"var-run-calico"},{"hostPath":{"path":"/var/lib/calico"},"name":"var-lib-calico"},{"hostPath":{"path":"/opt/cni/bin"},"name":"cni-bin-dir"},{"hostPath":{"path":"/etc/cni/net.d"},"name":"cni-net-dir"}]}},"updateStrategy":{"rollingUpdate":{"maxUnavailable":1},"type":"RollingUpdate"}}}
creationTimestamp: 2018-09-20T15:27:51Z
generation: 3
labels:
k8s-app: calico-node
name: calico-node
namespace: kube-system
resourceVersion: "319787"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/calico-node
uid: bc8d2642-bce9-11e8-ac07-0210da4b9e32
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: calico-node
template:
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
k8s-app: calico-node
spec:
containers:
- env:
- name: DATASTORE_TYPE
value: kubernetes
- name: FELIX_TYPHAK8SSERVICENAME
valueFrom:
configMapKeyRef:
key: typha_service_name
name: calico-config
- name: WAIT_FOR_DATASTORE
value: "true"
- name: NODENAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CALICO_NETWORKING_BACKEND
valueFrom:
configMapKeyRef:
key: calico_backend
name: calico-config
- name: CLUSTER_TYPE
value: k8s,bgp
- name: IP
value: autodetect
- name: CALICO_IPV4POOL_IPIP
value: Always
- name: FELIX_IPINIPENABLED
value: "true"
- name: FELIX_IPINIPMTU
value: "8981"
- name: CALICO_IPV4POOL_CIDR
value: 10.244.0.0/16
- name: CALICO_DISABLE_FILE_LOGGING
value: "true"
- name: FELIX_DEFAULTENDPOINTTOHOSTACTION
value: ACCEPT
- name: FELIX_IPV6SUPPORT
value: "false"
- name: FELIX_LOGSEVERITYSCREEN
value: info
- name: FELIX_HEALTHENABLED
value: "true"
image: quay.io/calico/node:v3.2.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 6
httpGet:
host: localhost
path: /liveness
port: 9099
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: calico-node
readinessProbe:
exec:
command:
- /bin/calico-node
- -bird-ready
- -felix-ready
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 250m
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /lib/modules
name: lib-modules
readOnly: true
- mountPath: /var/run/calico
name: var-run-calico
- mountPath: /var/lib/calico
name: var-lib-calico
- command:
- /install-cni.sh
env:
- name: CNI_CONF_NAME
value: 10-calico.conflist
- name: KUBERNETES_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CNI_NETWORK_CONFIG
valueFrom:
configMapKeyRef:
key: cni_network_config
name: calico-config
- name: CNI_MTU
valueFrom:
configMapKeyRef:
key: veth_mtu
name: calico-config
image: quay.io/calico/cni:v3.2.1
imagePullPolicy: IfNotPresent
name: install-cni
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
dnsPolicy: ClusterFirst
hostNetwork: true
nodeSelector:
beta.kubernetes.io/os: linux
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: calico-node
serviceAccountName: calico-node
terminationGracePeriodSeconds: 0
tolerations:
- effect: NoSchedule
operator: Exists
- key: CriticalAddonsOnly
operator: Exists
- effect: NoExecute
operator: Exists
volumes:
- hostPath:
path: /lib/modules
type: ""
name: lib-modules
- hostPath:
path: /var/run/calico
type: ""
name: var-run-calico
- hostPath:
path: /var/lib/calico
type: ""
name: var-lib-calico
- hostPath:
path: /opt/cni/bin
type: ""
name: cni-bin-dir
- hostPath:
path: /etc/cni/net.d
type: ""
name: cni-net-dir
templateGeneration: 3
updateStrategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
status:
currentNumberScheduled: 3
desiredNumberScheduled: 3
numberAvailable: 3
numberMisscheduled: 0
numberReady: 3
observedGeneration: 3
updatedNumberScheduled: 3

Calico-node Log getting variables

2018-09-20 16:51:22.636 [INFO][49] env_var_loader.go 40: Found felix environment variable: “ipinipmtu”=“8981"
2018-09-20 16:51:22.636 [INFO][49] config_params.go 217: Merging in config from environment variable: map[ipinipenabled:true typhak8sservicename:none logseverityscreen:info healthenabled:true etcdscheme: etcdendpoints: felixhostname:ip-172-31-0-160 etcdaddr: datastoretype:kubernetes etcdkeyfile: etcdcafile: ipv6support:false etcdcertfile: defaultendpointtohostaction:ACCEPT ipinipmtu:8981]

Tunl0 interface

265: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.244.1.1/32 brd 10.244.1.1 scope global tunl0
       valid_lft forever preferred_lft forever

Steps to Reproduce (for bugs)

  1. Bring up a cluster with Calico 3.2.1 (Have verified on RHEL, Ubuntu with Docker and Containerd)
  2. Install the standard kubernetes datastore method.
  3. Edit the daemonset and configmap to use 8981 as the mtu
  4. Verify in ifconfig that the tunl is updated
  5. Verify that the Calico interfaces are left at an mtu of 1500

Context

Trying to increase throughput of a cluster using jumbo frames within AWS.

Your Environment

  • Calico version - 3.2.1
  • Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes
  • Operating System and version: Ubuntu 16.04 and RHEL 7.5

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 15 (13 by maintainers)

Most upvoted comments

Given this is a CRI issue, I’ve raised https://github.com/containerd/cri/issues/981 against containerd, so I’m going to close this for now.

+1
caseydavenport on Nov 27, 2018
Read more comments on GitHub
← calico: Memory leak in calico-kube-controllers - v3.20
calico: Network policy isolation between namespaces does not prevent access  →