cilium: Request fails in all nodes every 5 minutes and when k8s service metadata is updated.

Bug reports

Title

Hi again, Thank you for #4510. After that time, I have tested a lot and found some problems 😃.

I tested sending requests from pod to cluster IP. I monitored the result, but there were request fails without any controls. It irregularly happened in all nodes about every 5 minutes. In this case, the revision number of Kubernetes resource is increased. Furthermore, when I tried to update k8s service, there were always fails either. (Even I changed metadata’s annotations, labels…)

General Information

  • Cilium version (run cilium version) I have tested many versions of Cilium.

I see these versions have the problem. 1.2.0, 1.2.1, latest(https://github.com/cilium/cilium/commit/a270d52ef158371fa2be06db53df39866af0080c)

These versions are OK for this issue. 1.0.9 (sorry I didn’t test 1.1)

  • Kernel version (run uname -a) 4.16.14-1.20180611.el7.centos.x86_64

  • Orchestration system version in use (e.g. kubectl version, Mesos, …) kubernetes 1.11

  • Link to relevant artifacts (policies, deployments scripts, …)

simple nodejs app

nodejs_deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nodejs-simple-deployment
spec:
  replicas: 10 
  template:
    metadata:
      labels:
        app: nodejs-simple-server
    spec:
      containers:
      - name: nodejs-simple-server
        image: heroku/nodejs-hello-world 
        imagePullPolicy: Always
        ports:
        - containerPort: 8000

nodejs_simple_service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nodejs-simple-server 
  labels:
    app: nodejs-simple-server
spec:
  selector:
    app: nodejs-simple-server
  ports:
  - port: 3000 
    protocol: TCP

locust_value.yaml

Name: locust
image:
  repository: quay.io/honestbee/locust
  tag: 0.7.5
  pullPolicy: IfNotPresent
  
  
service:
  name: master-web
  type: NodePort
  externalPort: 8089
  internalPort: 8089
  nodePort: 0
  annotations: {}
  extraLabels: {}
  
master:
  config:
    target-host: http://10.108.193.145:3000
  resources:
    limits:
      cpu: 100m
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi
  
  
worker:
  config:
    # all files from tasks folder are mounted under `/locust-tasks`
    locust-script: "/locust-tasks/tasks.py"

/charts/locust/tasks/task.py

from locust import HttpLocust, TaskSet, task

class ElbTasks(TaskSet):
  @task
  def status(self):
      self.client.get("/")

class ElbWarmer(HttpLocust):
  task_set = ElbTasks
  min_wait = 1000
  max_wait = 3000

locust -> nodejs_simple_service(cluster-ip:10.108.193.145:3000) request test

test app(locust): https://github.com/helm/charts/tree/master/stable/locust

How to reproduce the issue

  1. Create simple app (in my case, nodejs) nodejs_deployment.yaml nodejs_simple_service.yaml
$ kubectl apply -f nodejs_deployment.yaml
$ kubectl apply -f nodejs_simple_service.yaml
$ kubectl get svc
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
kubernetes                 ClusterIP   10.96.0.1        <none>        443/TCP                                        9h
nodejs-simple-server       ClusterIP   10.108.193.145   <none>        3000/TCP                                       21m

# service cluster ip: 10.108.193.145
  1. Create locust (https://github.com/helm/charts/tree/master/stable/locust)
helm install --name nodejs-tester -f locust_values.yaml <locust helm chart location>
# please check /charts/locust/tasks/task.py

  1. Send requests using locust to clusterIP (10.108.193.145)

  2. You can see request fails about every 5 minutes(little irregular)

  3. In addition, there is a definite fails with updating metadata while sending HTTP requests.

$ kubectl edit svc nodejs-simple-server

I have some ideas for these symptom. I assume that Service watch API of k8s with --rsync always updates all Service themselves every 5 minutes by k8s, and it may affect Cilium because Cilium may be also updating at that time.

$ cilium status --all-controllers 
KVStore:                Ok   etcd: 3/3 connected: http://10.127.146.91:2379 - 3.2.22 (Leader); http://10.127.157.99:2379 - 3.2.22; http://10.127.158.106:2379 - 3.2.22
ContainerRuntime:       Ok   containerD events watcher: Ok; cri-containerd client: Ok - &StatusResponse{Status:&RuntimeStatus{Conditions:[&RuntimeCondition{Type:RuntimeReady,Status:true,Reason:,Message:,} &RuntimeCondition{Type:NetworkReady,Status:true,Reason:,Message:,}],},Info:map[string]string{},}
Kubernetes:             Ok   1.11 (v1.11.2) [linux/amd64]
Kubernetes APIs:        ["core/v1::Endpoint", "core/v1::Pods", "core/v1::Node", "core/v1::Namespace", "CustomResourceDefinition", "cilium/v2::CiliumNetworkPolicy", "networking.k8s.io/v1::NetworkPolicy", "core/v1::Service"]
Cilium:                 Ok   OK
NodeMonitor:            Disabled
Cilium health daemon:   Ok   
IPv4 address pool:      5/127 allocated
IPv6 address pool:      4/65535 allocated
Controller Status:      24/24 healthy
  Name                                                                   Last success   Last error   Count   Message
  cilium-health-ep                                                       12s ago        never        0       no error   
  dns-poller                                                             5s ago         never        0       no error   
  etcd-lease-keepalive-0xc4215a4dc0                                      1m17s ago      never        0       no error   
  ipcache-bpf-garbage-collection                                         1m15s ago      never        0       no error   
  kvstore-etcd-session-renew                                             never          never        0       no error   
  kvstore-sync-store-cilium/state/nodes/v1                               16s ago        never        0       no error   
  lxcmap-bpf-host-sync                                                   5s ago         never        0       no error   
  metricsmap-bpf-prom-sync                                               4s ago         never        0       no error   
  resolve-identity-21092                                                 1m13s ago      never        0       no error   
  sync-IPv4-identity-mapping (14120)                                     1m13s ago      never        0       no error   
  sync-IPv4-identity-mapping (21092)                                     1m13s ago      never        0       no error   
  sync-IPv4-identity-mapping (40104)                                     1m13s ago      never        0       no error   
  sync-IPv6-identity-mapping (14120)                                     1m13s ago      never        0       no error   
  sync-IPv6-identity-mapping (21092)                                     1m13s ago      never        0       no error   
  sync-IPv6-identity-mapping (40104)                                     1m13s ago      never        0       no error   
  sync-identity-to-k8s-pod (14120)                                       12s ago        never        0       no error   
  sync-identity-to-k8s-pod (21092)                                       13s ago        never        0       no error   
  sync-identity-to-k8s-pod (40104)                                       13s ago        never        0       no error   
  sync-policymap-14120                                                   13s ago        never        0       no error   
  sync-policymap-21092                                                   13s ago        never        0       no error   
  sync-policymap-40104                                                   13s ago        never        0       no error   
  sync-to-k8s-ciliumendpoint (14120)                                     8s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (40104)                                     8s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint-gc (kube-ing-node003-nucleo-al7-jp2v-dev)   12s ago        never        0       no error   
Proxy Status:   OK, ip 192.168.5.1, port-range 10000-20000
Cluster health:   11/11 reachable   (2018-08-30T02:41:05Z)

  sync-IPv4-identity-mapping (14120)                                     1m13s ago      never        0       no error   
  sync-IPv4-identity-mapping (21092)                                     1m13s ago      never        0       no error   
  sync-IPv4-identity-mapping (40104)                                     1m13s ago      never        0       no error

I’m not sure that the endpoints of sync-IPv4-identity-mapping may work for something… when fails occur.

I attach the failure log below when I updated a metadata of Service. (Log in rsync update situation is too much…)

2018-08-30T10:51:26.251674603+09:00 level=debug msg="Received service update" k8sApiVersion= k8sNamespace=default k8sNamespace.old=default k8sSvcName=nodejs-simple-server k8sSvcName.old=nodejs-simple-server k8sSvcType=ClusterIP k8sSvcType.old=ClusterIP subsys=daemon
2018-08-30T10:51:26.25219497+09:00 level=debug msg="created new L3n4Addr" ipAddr="{10.108.193.145 {TCP 3000}}" subsys=loadbalancer
2018-08-30T10:51:26.252481502+09:00 level=debug msg="Resolving service" l3n4Addr="{IP:10.108.193.145 L4Addr:{Protocol:TCP Port:3000}}" subsys=service
2018-08-30T10:51:26.252546353+09:00 level=debug msg="Got feAddr ID for service" k8sNamespace=default k8sSvcName=nodejs-simple-server obj="{ServiceName:nodejs-simple-server Namespace:default}" serviceID=28 serviceName= subsys=daemon
2018-08-30T10:51:26.253541481+09:00 level=debug msg="created new L3n4Addr" ipAddr="{10.108.193.145 {TCP 3000}}" subsys=loadbalancer
2018-08-30T10:51:26.2544059+09:00 level=debug msg="adding service" obj="[{L3n4Addr:{IP:192.168.2.20 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.226 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.246 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.110 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4A
ddr:{IP:192.168.3.200 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.79 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.27 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.201 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.208 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:1
92.168.3.58 L4Addr:{Protocol:TCP Port:3000}} Weight:0}]" serviceID="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.255550825+09:00 level=debug msg="converting Cilium load-balancer service (frontend -> backend(s)) into BPF service" lbBackend="[{{192.168.2.20 {TCP 3000}} 0} {{192.168.2.226 {TCP 3000}} 0} {{192.168.2.246 {TCP 3000}} 0} {{192.168.3.110 {TCP 3000}} 0} {{192.168.3.200 {TCP 3000}} 0} {{192.168.3.79 {TCP 3000}} 0} {{192.168.2.27 {TCP 3000}} 0}
{{192.168.3.201 {TCP 3000}} 0} {{192.168.3.208 {TCP 3000}} 0} {{192.168.3.58 {TCP 3000}} 0}]" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.255590375+09:00 level=debug msg="converting L3n4Addr to ServiceKey" l3n4AddrID="{{10.108.193.145 {TCP 3000}} 28}" subsys=map-lb
2018-08-30T10:51:26.25665206+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.20:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.257316963+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.226:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.257938189+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.246:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.258599388+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.110:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.258627241+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.200:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.258668114+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.79:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.258951796+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.27:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.260034233+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.201:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.26018491+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.208:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.261117951+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.58:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.261236444+09:00 level=debug msg="converted LBSVC (frontend -> backend(s)), to Service Key and Value" lbBackend="[{{192.168.2.20 {TCP 3000}} 0} {{192.168.2.226 {TCP 3000}} 0} {{192.168.2.246 {TCP 3000}} 0} {{192.168.3.110 {TCP 3000}} 0} {{192.168.3.200 {TCP 3000}} 0} {{192.168.3.79 {TCP 3000}} 0} {{192.168.2.27 {TCP 3000}} 0} {{192.168.3.201
 {TCP 3000}} 0} {{192.168.3.208 {TCP 3000}} 0} {{192.168.3.58 {TCP 3000}} 0}]" lbFrontend="10.108.193.145:3000" obj="[192.168.2.20:3000 (28) 192.168.2.226:3000 (28) 192.168.2.246:3000 (28) 192.168.3.110:3000 (28) 192.168.3.200:3000 (28) 192.168.3.79:3000 (28) 192.168.2.27:3000 (28) 192.168.3.201:3000 (28) 192.168.3.208:3000 (28) 192.168.3.58:3000 (28)]" servic
eID="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.261605181+09:00 level=debug msg="adding service to BPF maps" serviceName="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.261704926+09:00 level=debug msg="deleting service from BPF maps" serviceName="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.262970532+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=10 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.263020875+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=9 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.263190085+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=8 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.263214564+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=7 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.264299165+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=6 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.264515412+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=5 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.264586494+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=4 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.266019287+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=3 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.266081834+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=2 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.266096899+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=1 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.266407345+09:00 level=debug msg="done deleting service slaves, now deleting master service" serviceID=28 subsys=daemon
2018-08-30T10:51:26.266426521+09:00 level=debug msg="deleting service from loadbalancer" serviceName="10.108.193.145:3000" sha=e3d018bbcc14ecb10372879557e2ddae5197339945947a94ac84eba3d03e2dd9 subsys=loadbalancer
2018-08-30T10:51:26.26750007+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.20:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.267731167+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.226:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.268145215+09:00 level=debug msg="Controller func execution time: 38.324922ms" name="sync-to-k8s-ciliumendpoint (54767)" subsys=controller uuid=b9ab5bbd-abf5-11e8-a21f-fa163eed3276
2018-08-30T10:51:26.268186103+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.246:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.268764817+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.110:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.269315069+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.200:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270186567+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.79:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270231496+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.27:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270247677+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.201:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270285331+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.208:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270294253+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.58:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270300317+09:00 level=debug msg="adding revNat to lbmap" bpfMapKey=28 bpfMapValue="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270312506+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="0.0.0.0:0 (0)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.270319363+09:00 level=debug msg="adding service to RevNATMap" serviceName="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.270326234+09:00 level=debug msg="adding service to loadbalancer" serviceName="10.108.193.145:3000" sha=e3d018bbcc14ecb10372879557e2ddae5197339945947a94ac84eba3d03e2dd9 subsys=loadbalancer
2018-08-30T10:51:26.338106017+09:00 level=debug msg="created new L3n4Addr" ipAddr="{10.108.193.145 {TCP 3000}}" subsys=loadbalancer
2018-08-30T10:51:26.338195762+09:00 level=debug msg="adding service" obj="[{L3n4Addr:{IP:192.168.2.226 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.201 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.208 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.20 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n
4Addr:{IP:192.168.2.246 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.27 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.110 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.200 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.58 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP
:192.168.3.79 L4Addr:{Protocol:TCP Port:3000}} Weight:0}]" serviceID="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.338195762+09:00 level=debug msg="adding service" obj="[{L3n4Addr:{IP:192.168.2.226 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.201 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.208 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.20 L4Addr:{Protocol:TCP Port:3000}} Weigh[61/1103]
4Addr:{IP:192.168.2.246 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.2.27 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.110 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.200 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP:192.168.3.58 L4Addr:{Protocol:TCP Port:3000}} Weight:0} {L3n4Addr:{IP
:192.168.3.79 L4Addr:{Protocol:TCP Port:3000}} Weight:0}]" serviceID="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.339749706+09:00 level=debug msg="converting Cilium load-balancer service (frontend -> backend(s)) into BPF service" lbBackend="[{{192.168.2.226 {TCP 3000}} 0} {{192.168.3.201 {TCP 3000}} 0} {{192.168.3.208 {TCP 3000}} 0} {{192.168.2.20 {TCP 3000}} 0} {{192.168.2.246 {TCP 3000}} 0} {{192.168.2.27 {TCP 3000}} 0} {{192.168.3.110 {TCP 3000}} 0}
 {{192.168.3.200 {TCP 3000}} 0} {{192.168.3.58 {TCP 3000}} 0} {{192.168.3.79 {TCP 3000}} 0}]" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.339788476+09:00 level=debug msg="converting L3n4Addr to ServiceKey" l3n4AddrID="{{10.108.193.145 {TCP 3000}} 28}" subsys=map-lb
2018-08-30T10:51:26.340858808+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.226:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.340878376+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.201:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.340947532+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.208:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.341053407+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.20:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.342233022+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.246:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.342456951+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.2.27:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.343052373+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.110:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.343225834+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.200:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.34425476+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.58:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.344278839+09:00 level=debug msg="associating frontend -> backend" lbBackend="192.168.3.79:3000 (28)" lbFrontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.345098522+09:00 level=debug msg="converted LBSVC (frontend -> backend(s)), to Service Key and Value" lbBackend="[{{192.168.2.226 {TCP 3000}} 0} {{192.168.3.201 {TCP 3000}} 0} {{192.168.3.208 {TCP 3000}} 0} {{192.168.2.20 {TCP 3000}} 0} {{192.168.2.246 {TCP 3000}} 0} {{192.168.2.27 {TCP 3000}} 0} {{192.168.3.110 {TCP 3000}} 0} {{192.168.3.20
0 {TCP 3000}} 0} {{192.168.3.58 {TCP 3000}} 0} {{192.168.3.79 {TCP 3000}} 0}]" lbFrontend="10.108.193.145:3000" obj="[192.168.2.226:3000 (28) 192.168.3.201:3000 (28) 192.168.3.208:3000 (28) 192.168.2.20:3000 (28) 192.168.2.246:3000 (28) 192.168.2.27:3000 (28) 192.168.3.110:3000 (28) 192.168.3.200:3000 (28) 192.168.3.58:3000 (28) 192.168.3.79:3000 (28)]" servic
eID="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.345132353+09:00 level=debug msg="adding service to BPF maps" serviceName="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.345254135+09:00 level=debug msg="deleting service from BPF maps" serviceName="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.345344651+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=10 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.346302941+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=9 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.346445159+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=8 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.347350062+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=7 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.347397172+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=6 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.347771784+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=5 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.347927173+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=4 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.348671538+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=3 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.348726393+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=2 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.348751481+09:00 level=debug msg="deleting backend # for slave ServiceKey" idx.backend=1 key="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.348989148+09:00 level=debug msg="done deleting service slaves, now deleting master service" serviceID=28 subsys=daemon
2018-08-30T10:51:26.349724567+09:00 level=debug msg="deleting service from loadbalancer" serviceName="10.108.193.145:3000" sha=e3d018bbcc14ecb10372879557e2ddae5197339945947a94ac84eba3d03e2dd9 subsys=loadbalancer
2018-08-30T10:51:26.349749492+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.226:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349755895+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.201:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349780394+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.208:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349785501+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.20:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349795768+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.246:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349851317+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.2.27:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349857436+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.110:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349866389+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.200:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.3498754+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.58:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349913218+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="192.168.3.79:3000 (28)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349936102+09:00 level=debug msg="adding revNat to lbmap" bpfMapKey=28 bpfMapValue="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.349945553+09:00 level=debug msg="adding frontend for backend to BPF maps" backend="0.0.0.0:0 (0)" frontend="10.108.193.145:3000" subsys=map-lb
2018-08-30T10:51:26.34997483+09:00 level=debug msg="adding service to RevNATMap" serviceName="10.108.193.145:3000" subsys=daemon
2018-08-30T10:51:26.349985865+09:00 level=debug msg="adding service to loadbalancer" serviceName="10.108.193.145:3000" sha=e3d018bbcc14ecb10372879557e2ddae5197339945947a94ac84eba3d03e2dd9 subsys=loadbalancer

Thank you for your fast reply last time. I always expect Cilium and ready for testing and using it! If you have any question or need more tests, then please tell me anytime.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 2
  • Comments: 16 (13 by maintainers)

Commits related to this issue

Most upvoted comments

I checked all the existing problems for a few hours and confirmed that cilium/cilium-dev:v1.2.2-rc2 had no problems at all.

Awesome. Thanks for testing!

+3
tgraf on Sep 8, 2018

@aanm @tgraf @ianvernon

Hi, first of all, thanks for great project. I’m testing it with @eldin77. We’re running a test with https://github.com/cilium/cilium/commit/05e99f3124e81e1c697a547327908bcb6c33c5d8 commit, and it contains #5469, #5475. As we continue to test with locust, I think the problem is narrowing down overall so I want to share our updated testing result.

Unfortunately, still connection failures occur every five minutes. The connection fail message looks like this:

GET /   "ConnectionError(ProtocolError('Connection aborted.', error(104, 'Connection reset by peer')),)"

But now I think the k8s service related problem is solved. I mean below thing was a problem, but now the problem is solved.

  • Service resync from k8s every 5 minutes (guessing troubleshooted by #5475 )
    • I am modifying the metadata.annotations of the service by hand to reproduce this. We are testing this assuming that the load balancer content(e.g. feip, feport, beip, beport) does not change but service updated. That is, it is a method to call the following function without changing the service. But there is no connection failure anymore with this.
    func (d *Daemon) addK8sServiceV1(svc *v1.Service) {

But I do not have a problem with the service, so the other problem seems to be clear. Previously, it seemed that service resync and endpoint resync were executed at the same time and could not be distinguished. I think the problem below is left.

  • Endpoint resync from k8s every 5 minutes
    • I think if you modify the endpoint’s metadata.annotations like above, you will be able to reproduce the same connection failures by hands.

I hope this comment helps solve this problem. Thank you very much.

+3
leoh0 on Sep 6, 2018

@leoh0 @eldin77 #5502 is providing a fix for the issue. We are testing this with a streaming connection while updating the annotations in parallel. Would be great if you could confirm that this fixes the issue for you as well. We have built a custom dev image cilium/cilium-dev:v1.2.2-rc2 to include this fix.

+2
tgraf on Sep 8, 2018

@tgraf

I attached a link to the results generated when I updated the metadata.annotations on the k8s nodejs-simple-server endpoint. And currently all worker requests HTTP GET to http://nodejs-simple-server.pt.svc.cluster.local:3000 (which is 10.108.27.179:3000).

I think you can check the RST packets below cilium monitor gist link.

# grep only RST packaets
<- endpoint 25104 flow 0xbfed203 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.225:48104 tcp RST
-> endpoint 48734 flow 0xbfed203 identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:48104 tcp RST
<- endpoint 25104 flow 0x37cb2274 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.161:42362 tcp RST
-> endpoint 12289 flow 0x37cb2274 identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:42362 tcp RST
<- endpoint 35597 flow 0xcaad7e3d identity 3783->0 state new ifindex 0: 192.168.1.185:3000 -> 192.168.1.193:60364 tcp RST
-> endpoint 11557 flow 0xcaad7e3d identity 3783->42518 state reply ifindex lxc67dfb: 10.108.27.179:3000 -> 192.168.1.193:60364 tcp RST
<- endpoint 33229 flow 0xd454439a identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.165:54778 tcp RST
-> endpoint 37905 flow 0xd454439a identity 3783->42518 state reply ifindex lxc864e6: 10.108.27.179:3000 -> 192.168.1.165:54778 tcp RST
<- endpoint 26549 flow 0xf303e3c2 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.152:39438 tcp RST
-> endpoint 41397 flow 0xf303e3c2 identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:39438 tcp RST
<- endpoint 33229 flow 0x87f1762c identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.174:45332 tcp RST
-> endpoint 60566 flow 0x87f1762c identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:45332 tcp RST
<- endpoint 33229 flow 0x18438aed identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.152:37932 tcp RST
-> endpoint 41397 flow 0x18438aed identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:37932 tcp RST
<- endpoint 5751 flow 0x7393d58b identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.165:54700 tcp RST
-> endpoint 37905 flow 0x7393d58b identity 3783->42518 state reply ifindex lxc864e6: 10.108.27.179:3000 -> 192.168.1.165:54700 tcp RST
<- endpoint 25104 flow 0xef37017 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.225:47036 tcp RST
-> endpoint 48734 flow 0xef37017 identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:47036 tcp RST
<- endpoint 65430 flow 0xa5912e3d identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.152:39462 tcp RST
-> endpoint 41397 flow 0xa5912e3d identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:39462 tcp RST
<- endpoint 33229 flow 0x3f2e01ff identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.205:51188 tcp RST
-> endpoint 49212 flow 0x3f2e01ff identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:51188 tcp RST
<- endpoint 5751 flow 0xa96717e0 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.161:42126 tcp RST
-> endpoint 12289 flow 0xa96717e0 identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:42126 tcp RST
<- endpoint 65430 flow 0xa7033577 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.174:45224 tcp RST
-> endpoint 60566 flow 0xa7033577 identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:45224 tcp RST
<- endpoint 25104 flow 0x27ec75b9 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.176:53084 tcp RST
-> endpoint 24349 flow 0x27ec75b9 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:53084 tcp RST
<- endpoint 35597 flow 0x8ca4ad80 identity 3783->0 state new ifindex 0: 192.168.1.185:3000 -> 192.168.1.197:44302 tcp RST
-> endpoint 27636 flow 0x8ca4ad80 identity 3783->42518 state reply ifindex lxc51684: 10.108.27.179:3000 -> 192.168.1.197:44302 tcp RST
<- endpoint 40784 flow 0x764b5e40 identity 3783->0 state new ifindex 0: 192.168.1.229:3000 -> 192.168.1.165:55888 tcp RST
-> endpoint 37905 flow 0x764b5e40 identity 3783->42518 state reply ifindex lxc864e6: 10.108.27.179:3000 -> 192.168.1.165:55888 tcp RST
<- endpoint 5751 flow 0xa43223a4 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.137:43054 tcp RST
-> endpoint 2350 flow 0xa43223a4 identity 3783->42518 state reply ifindex lxcbddb9: 10.108.27.179:3000 -> 192.168.1.137:43054 tcp RST
<- endpoint 25104 flow 0x98ebe8c4 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.170:41868 tcp RST
-> endpoint 41841 flow 0x98ebe8c4 identity 3783->42518 state reply ifindex lxce1d2c: 10.108.27.179:3000 -> 192.168.1.170:41868 tcp RST
<- endpoint 25104 flow 0xc16810fc identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.236:58354 tcp RST
-> endpoint 4993 flow 0xc16810fc identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:58354 tcp RST
<- endpoint 65430 flow 0xa84bf25a identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.152:38324 tcp RST
-> endpoint 41397 flow 0xa84bf25a identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:38324 tcp RST
<- endpoint 33229 flow 0x9905ea6c identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.168:43124 tcp RST
-> endpoint 28921 flow 0x9905ea6c identity 3783->42518 state reply ifindex lxcca58a: 10.108.27.179:3000 -> 192.168.1.168:43124 tcp RST
<- endpoint 26549 flow 0x462cd52e identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.236:59378 tcp RST
-> endpoint 4993 flow 0x462cd52e identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:59378 tcp RST
<- endpoint 50944 flow 0xda2ff76c identity 3783->0 state new ifindex 0: 192.168.1.169:3000 -> 192.168.1.152:39398 tcp RST
-> endpoint 41397 flow 0xda2ff76c identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:39398 tcp RST
<- endpoint 25104 flow 0x9166d8d8 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.168:41726 tcp RST
-> endpoint 28921 flow 0x9166d8d8 identity 3783->42518 state reply ifindex lxcca58a: 10.108.27.179:3000 -> 192.168.1.168:41726 tcp RST
<- endpoint 33229 flow 0xa941683 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.205:52372 tcp RST
-> endpoint 49212 flow 0xa941683 identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:52372 tcp RST
<- endpoint 33229 flow 0x2be02dde identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.170:40786 tcp RST
-> endpoint 41841 flow 0x2be02dde identity 3783->42518 state reply ifindex lxce1d2c: 10.108.27.179:3000 -> 192.168.1.170:40786 tcp RST
<- endpoint 33229 flow 0x6ebc6dca identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.174:45136 tcp RST
-> endpoint 60566 flow 0x6ebc6dca identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:45136 tcp RST
<- endpoint 26549 flow 0x866ec271 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.173:45014 tcp RST
-> endpoint 37521 flow 0x866ec271 identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:45014 tcp RST
<- endpoint 65430 flow 0xc75adc37 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.161:42214 tcp RST
-> endpoint 12289 flow 0xc75adc37 identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:42214 tcp RST
<- endpoint 65430 flow 0x4223e16a identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.197:43216 tcp RST
-> endpoint 27636 flow 0x4223e16a identity 3783->42518 state reply ifindex lxc51684: 10.108.27.179:3000 -> 192.168.1.197:43216 tcp RST
<- endpoint 25104 flow 0x2042f344 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.161:42282 tcp RST
-> endpoint 12289 flow 0x2042f344 identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:42282 tcp RST
<- endpoint 65430 flow 0xc7b767e4 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.188:34564 tcp RST
-> endpoint 11201 flow 0xc7b767e4 identity 3783->42518 state reply ifindex lxc57835: 10.108.27.179:3000 -> 192.168.1.188:34564 tcp RST
<- endpoint 5751 flow 0xfe1adbbe identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.170:40744 tcp RST
-> endpoint 41841 flow 0xfe1adbbe identity 3783->42518 state reply ifindex lxce1d2c: 10.108.27.179:3000 -> 192.168.1.170:40744 tcp RST
<- endpoint 65430 flow 0x49c2b0f1 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.171:59620 tcp RST
-> endpoint 21162 flow 0x49c2b0f1 identity 3783->42518 state reply ifindex lxcdf014: 10.108.27.179:3000 -> 192.168.1.171:59620 tcp RST
<- endpoint 5751 flow 0xa39ac72d identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.152:38246 tcp RST
-> endpoint 41397 flow 0xa39ac72d identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:38246 tcp RST
<- endpoint 27030 flow 0x63a1f79e identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.171:58564 tcp RST
-> endpoint 21162 flow 0x63a1f79e identity 3783->42518 state reply ifindex lxcdf014: 10.108.27.179:3000 -> 192.168.1.171:58564 tcp RST
<- endpoint 65430 flow 0x6b4eb533 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.161:42446 tcp RST
-> endpoint 12289 flow 0x6b4eb533 identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:42446 tcp RST
<- endpoint 33229 flow 0xe70b1108 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.174:45168 tcp RST
-> endpoint 60566 flow 0xe70b1108 identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:45168 tcp RST
<- endpoint 27030 flow 0xa5e33905 identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.204:57342 tcp RST
-> endpoint 45277 flow 0xa5e33905 identity 3783->42518 state reply ifindex lxc15a64: 10.108.27.179:3000 -> 192.168.1.204:57342 tcp RST
<- endpoint 33229 flow 0x5e6b822 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.204:56078 tcp RST
-> endpoint 45277 flow 0x5e6b822 identity 3783->42518 state reply ifindex lxc15a64: 10.108.27.179:3000 -> 192.168.1.204:56078 tcp RST
<- endpoint 25104 flow 0xf55f5d73 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.174:45260 tcp RST
-> endpoint 60566 flow 0xf55f5d73 identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:45260 tcp RST
<- endpoint 27030 flow 0x4e77c3d9 identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.197:42966 tcp RST
-> endpoint 27636 flow 0x4e77c3d9 identity 3783->42518 state reply ifindex lxc51684: 10.108.27.179:3000 -> 192.168.1.197:42966 tcp RST
<- endpoint 26549 flow 0xe5978e49 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.174:46360 tcp RST
-> endpoint 60566 flow 0xe5978e49 identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:46360 tcp RST
<- endpoint 50944 flow 0x5231d3a8 identity 3783->0 state new ifindex 0: 192.168.1.169:3000 -> 192.168.1.161:43478 tcp RST
-> endpoint 12289 flow 0x5231d3a8 identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:43478 tcp RST
<- endpoint 65430 flow 0xedbec06 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.170:41806 tcp RST
-> endpoint 41841 flow 0xedbec06 identity 3783->42518 state reply ifindex lxce1d2c: 10.108.27.179:3000 -> 192.168.1.170:41806 tcp RST
<- endpoint 25104 flow 0x4cfa08db identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.137:43074 tcp RST
-> endpoint 2350 flow 0x4cfa08db identity 3783->42518 state reply ifindex lxcbddb9: 10.108.27.179:3000 -> 192.168.1.137:43074 tcp RST
<- endpoint 33229 flow 0xbfb4226e identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.174:46564 tcp RST
-> endpoint 60566 flow 0xbfb4226e identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:46564 tcp RST
<- endpoint 5751 flow 0xe449ce07 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.171:58544 tcp RST
-> endpoint 21162 flow 0xe449ce07 identity 3783->42518 state reply ifindex lxcdf014: 10.108.27.179:3000 -> 192.168.1.171:58544 tcp RST
<- endpoint 65430 flow 0xc3f0bdf0 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.193:60528 tcp RST
-> endpoint 11557 flow 0xc3f0bdf0 identity 3783->42518 state reply ifindex lxc67dfb: 10.108.27.179:3000 -> 192.168.1.193:60528 tcp RST
<- endpoint 33229 flow 0xb93fa32a identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.152:38280 tcp RST
-> endpoint 41397 flow 0xb93fa32a identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:38280 tcp RST
<- endpoint 25104 flow 0xe44a2c07 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.176:52990 tcp RST
-> endpoint 24349 flow 0xe44a2c07 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:52990 tcp RST
<- endpoint 65430 flow 0x64683e9a identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.225:48024 tcp RST
-> endpoint 48734 flow 0x64683e9a identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:48024 tcp RST
<- endpoint 5751 flow 0xb0813427 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.173:44954 tcp RST
-> endpoint 37521 flow 0xb0813427 identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:44954 tcp RST
<- endpoint 25104 flow 0xc578cf0c identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.205:52370 tcp RST
-> endpoint 49212 flow 0xc578cf0c identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:52370 tcp RST
<- endpoint 26549 flow 0x5e30c7c3 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.176:54184 tcp RST
-> endpoint 24349 flow 0x5e30c7c3 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:54184 tcp RST
<- endpoint 27030 flow 0x1f7e3f0d identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.232:45134 tcp RST
-> endpoint 654 flow 0x1f7e3f0d identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:45134 tcp RST
<- endpoint 25104 flow 0x462430c8 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.225:47028 tcp RST
-> endpoint 48734 flow 0x462430c8 identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:47028 tcp RST
<- endpoint 50944 flow 0x2812feb8 identity 3783->0 state new ifindex 0: 192.168.1.169:3000 -> 192.168.1.173:44884 tcp RST
-> endpoint 37521 flow 0x2812feb8 identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:44884 tcp RST
<- endpoint 50944 flow 0x2356d002 identity 3783->0 state new ifindex 0: 192.168.1.169:3000 -> 192.168.1.138:51424 tcp RST
-> endpoint 62268 flow 0x2356d002 identity 3783->42518 state reply ifindex lxc28d65: 10.108.27.179:3000 -> 192.168.1.138:51424 tcp RST
<- endpoint 25104 flow 0xe482c195 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.236:58262 tcp RST
-> endpoint 4993 flow 0xe482c195 identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:58262 tcp RST
<- endpoint 25104 flow 0xd99c1e8c identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.165:54752 tcp RST
-> endpoint 37905 flow 0xd99c1e8c identity 3783->42518 state reply ifindex lxc864e6: 10.108.27.179:3000 -> 192.168.1.165:54752 tcp RST
<- endpoint 5751 flow 0xb1d7748f identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.232:44996 tcp RST
-> endpoint 654 flow 0xb1d7748f identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:44996 tcp RST
<- endpoint 33229 flow 0xdd2b54b6 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.137:44280 tcp RST
-> endpoint 2350 flow 0xdd2b54b6 identity 3783->42518 state reply ifindex lxcbddb9: 10.108.27.179:3000 -> 192.168.1.137:44280 tcp RST
<- endpoint 25104 flow 0x263d2691 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.168:41824 tcp RST
-> endpoint 28921 flow 0x263d2691 identity 3783->42518 state reply ifindex lxcca58a: 10.108.27.179:3000 -> 192.168.1.168:41824 tcp RST
<- endpoint 65430 flow 0xdfb30902 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.197:44220 tcp RST
-> endpoint 27636 flow 0xdfb30902 identity 3783->42518 state reply ifindex lxc51684: 10.108.27.179:3000 -> 192.168.1.197:44220 tcp RST
<- endpoint 50944 flow 0xef18bd48 identity 3783->0 state new ifindex 0: 192.168.1.169:3000 -> 192.168.1.176:54120 tcp RST
-> endpoint 24349 flow 0xef18bd48 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:54120 tcp RST
<- endpoint 27030 flow 0x9abba2db identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.161:42494 tcp RST
-> endpoint 12289 flow 0x9abba2db identity 3783->42518 state reply ifindex lxcb7fa5: 10.108.27.179:3000 -> 192.168.1.161:42494 tcp RST
<- endpoint 25104 flow 0xc8e60537 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.204:57404 tcp RST
-> endpoint 45277 flow 0xc8e60537 identity 3783->42518 state reply ifindex lxc15a64: 10.108.27.179:3000 -> 192.168.1.204:57404 tcp RST
<- endpoint 27030 flow 0xa8ba8b7e identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.204:56284 tcp RST
-> endpoint 45277 flow 0xa8ba8b7e identity 3783->42518 state reply ifindex lxc15a64: 10.108.27.179:3000 -> 192.168.1.204:56284 tcp RST
<- endpoint 27030 flow 0xfa473fee identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.173:43728 tcp RST
-> endpoint 37521 flow 0xfa473fee identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:43728 tcp RST
<- endpoint 33229 flow 0x811d8a98 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.248:36644 tcp RST
-> endpoint 59907 flow 0x811d8a98 identity 3783->42518 state reply ifindex lxcd057f: 10.108.27.179:3000 -> 192.168.1.248:36644 tcp RST
<- endpoint 5751 flow 0x7e373a34 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.197:42926 tcp RST
-> endpoint 27636 flow 0x7e373a34 identity 3783->42518 state reply ifindex lxc51684: 10.108.27.179:3000 -> 192.168.1.197:42926 tcp RST
<- endpoint 65430 flow 0x4ec4309 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.173:43752 tcp RST
-> endpoint 37521 flow 0x4ec4309 identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:43752 tcp RST
<- endpoint 26549 flow 0x9c2b6ea6 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.138:51248 tcp RST
-> endpoint 62268 flow 0x9c2b6ea6 identity 3783->42518 state reply ifindex lxc28d65: 10.108.27.179:3000 -> 192.168.1.138:51248 tcp RST
<- endpoint 25104 flow 0x1f0abacb identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.204:57510 tcp RST
-> endpoint 45277 flow 0x1f0abacb identity 3783->42518 state reply ifindex lxc15a64: 10.108.27.179:3000 -> 192.168.1.204:57510 tcp RST
<- endpoint 27030 flow 0x7d559436 identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.232:46202 tcp RST
-> endpoint 654 flow 0x7d559436 identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:46202 tcp RST
<- endpoint 5751 flow 0x6f2eac3b identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.188:33402 tcp RST
-> endpoint 11201 flow 0x6f2eac3b identity 3783->42518 state reply ifindex lxc57835: 10.108.27.179:3000 -> 192.168.1.188:33402 tcp RST
<- endpoint 26549 flow 0xeeeb8f7d identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.171:59704 tcp RST
-> endpoint 21162 flow 0xeeeb8f7d identity 3783->42518 state reply ifindex lxcdf014: 10.108.27.179:3000 -> 192.168.1.171:59704 tcp RST
<- endpoint 50944 flow 0xc9382a29 identity 3783->0 state new ifindex 0: 192.168.1.169:3000 -> 192.168.1.197:44394 tcp RST
-> endpoint 27636 flow 0xc9382a29 identity 3783->42518 state reply ifindex lxc51684: 10.108.27.179:3000 -> 192.168.1.197:44394 tcp RST
<- endpoint 25104 flow 0x32963574 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.188:33174 tcp RST
-> endpoint 11201 flow 0x32963574 identity 3783->42518 state reply ifindex lxc57835: 10.108.27.179:3000 -> 192.168.1.188:33174 tcp RST
<- endpoint 65430 flow 0xc09e1bd3 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.170:40602 tcp RST
-> endpoint 41841 flow 0xc09e1bd3 identity 3783->42518 state reply ifindex lxce1d2c: 10.108.27.179:3000 -> 192.168.1.170:40602 tcp RST
<- endpoint 25104 flow 0x9038559a identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.188:34470 tcp RST
-> endpoint 11201 flow 0x9038559a identity 3783->42518 state reply ifindex lxc57835: 10.108.27.179:3000 -> 192.168.1.188:34470 tcp RST
<- endpoint 33229 flow 0x58f72040 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.225:48110 tcp RST
-> endpoint 48734 flow 0x58f72040 identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:48110 tcp RST
<- endpoint 25104 flow 0x319bb67a identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.188:34594 tcp RST
-> endpoint 11201 flow 0x319bb67a identity 3783->42518 state reply ifindex lxc57835: 10.108.27.179:3000 -> 192.168.1.188:34594 tcp RST
<- endpoint 25104 flow 0xdf0a5c5a identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.176:54186 tcp RST
-> endpoint 24349 flow 0xdf0a5c5a identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:54186 tcp RST
<- endpoint 65430 flow 0xde044fb8 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.138:50026 tcp RST
-> endpoint 62268 flow 0xde044fb8 identity 3783->42518 state reply ifindex lxc28d65: 10.108.27.179:3000 -> 192.168.1.138:50026 tcp RST
<- endpoint 33229 flow 0x61206597 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.138:49948 tcp RST
-> endpoint 62268 flow 0x61206597 identity 3783->42518 state reply ifindex lxc28d65: 10.108.27.179:3000 -> 192.168.1.138:49948 tcp RST
<- endpoint 35597 flow 0xbb6bc5cf identity 3783->0 state new ifindex 0: 192.168.1.185:3000 -> 192.168.1.232:46298 tcp RST
-> endpoint 654 flow 0xbb6bc5cf identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:46298 tcp RST
<- endpoint 27030 flow 0xcea8655b identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.204:57418 tcp RST
-> endpoint 45277 flow 0xcea8655b identity 3783->42518 state reply ifindex lxc15a64: 10.108.27.179:3000 -> 192.168.1.204:57418 tcp RST
<- endpoint 33229 flow 0x83cc152a identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.188:34552 tcp RST
-> endpoint 11201 flow 0x83cc152a identity 3783->42518 state reply ifindex lxc57835: 10.108.27.179:3000 -> 192.168.1.188:34552 tcp RST
<- endpoint 65430 flow 0xc8246bba identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.176:54218 tcp RST
-> endpoint 24349 flow 0xc8246bba identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:54218 tcp RST
<- endpoint 27030 flow 0x99d5c59d identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.171:58542 tcp RST
-> endpoint 21162 flow 0x99d5c59d identity 3783->42518 state reply ifindex lxcdf014: 10.108.27.179:3000 -> 192.168.1.171:58542 tcp RST
<- endpoint 33229 flow 0x247fc125 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.174:46504 tcp RST
-> endpoint 60566 flow 0x247fc125 identity 3783->42518 state reply ifindex lxcba3f9: 10.108.27.179:3000 -> 192.168.1.174:46504 tcp RST
<- endpoint 25104 flow 0x978b3f40 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.168:41694 tcp RST
-> endpoint 28921 flow 0x978b3f40 identity 3783->42518 state reply ifindex lxcca58a: 10.108.27.179:3000 -> 192.168.1.168:41694 tcp RST
<- endpoint 5751 flow 0x55231c61 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.176:53104 tcp RST
-> endpoint 24349 flow 0x55231c61 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:53104 tcp RST
<- endpoint 27030 flow 0xedd15ce3 identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.170:40492 tcp RST
-> endpoint 41841 flow 0xedd15ce3 identity 3783->42518 state reply ifindex lxce1d2c: 10.108.27.179:3000 -> 192.168.1.170:40492 tcp RST
<- endpoint 26549 flow 0x98f55c85 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.137:44166 tcp RST
-> endpoint 2350 flow 0x98f55c85 identity 3783->42518 state reply ifindex lxcbddb9: 10.108.27.179:3000 -> 192.168.1.137:44166 tcp RST
<- endpoint 26549 flow 0xe87243b1 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.205:52542 tcp RST
-> endpoint 49212 flow 0xe87243b1 identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:52542 tcp RST
<- endpoint 33229 flow 0x43982d6c identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.152:39366 tcp RST
-> endpoint 41397 flow 0x43982d6c identity 3783->42518 state reply ifindex lxcade6f: 10.108.27.179:3000 -> 192.168.1.152:39366 tcp RST
<- endpoint 35597 flow 0xb6127c9c identity 3783->0 state new ifindex 0: 192.168.1.185:3000 -> 192.168.1.205:52422 tcp RST
-> endpoint 49212 flow 0xb6127c9c identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:52422 tcp RST
<- endpoint 35597 flow 0x9fc38d94 identity 3783->0 state new ifindex 0: 192.168.1.185:3000 -> 192.168.1.193:60380 tcp RST
-> endpoint 11557 flow 0x9fc38d94 identity 3783->42518 state reply ifindex lxc67dfb: 10.108.27.179:3000 -> 192.168.1.193:60380 tcp RST
<- endpoint 65430 flow 0xc7d53dd6 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.176:53000 tcp RST
-> endpoint 24349 flow 0xc7d53dd6 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:53000 tcp RST
<- endpoint 5751 flow 0x52171f68 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.232:45070 tcp RST
-> endpoint 654 flow 0x52171f68 identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:45070 tcp RST
<- endpoint 65430 flow 0xc7fdc421 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.248:35670 tcp RST
-> endpoint 59907 flow 0xc7fdc421 identity 3783->42518 state reply ifindex lxcd057f: 10.108.27.179:3000 -> 192.168.1.248:35670 tcp RST
<- endpoint 40784 flow 0x2cff01cb identity 3783->0 state new ifindex 0: 192.168.1.229:3000 -> 192.168.1.225:48082 tcp RST
-> endpoint 48734 flow 0x2cff01cb identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:48082 tcp RST
<- endpoint 25104 flow 0x206e2bcc identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.168:41666 tcp RST
-> endpoint 28921 flow 0x206e2bcc identity 3783->42518 state reply ifindex lxcca58a: 10.108.27.179:3000 -> 192.168.1.168:41666 tcp RST
<- endpoint 65430 flow 0x96a48e4 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.193:59208 tcp RST
-> endpoint 11557 flow 0x96a48e4 identity 3783->42518 state reply ifindex lxc67dfb: 10.108.27.179:3000 -> 192.168.1.193:59208 tcp RST
<- endpoint 5751 flow 0xbe7bdb71 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.236:58260 tcp RST
-> endpoint 4993 flow 0xbe7bdb71 identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:58260 tcp RST
<- endpoint 25104 flow 0xcd925867 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.176:52768 tcp RST
-> endpoint 24349 flow 0xcd925867 identity 3783->42518 state reply ifindex lxc5b9e1: 10.108.27.179:3000 -> 192.168.1.176:52768 tcp RST
<- endpoint 40784 flow 0x28d0843b identity 3783->0 state new ifindex 0: 192.168.1.229:3000 -> 192.168.1.171:59716 tcp RST
-> endpoint 21162 flow 0x28d0843b identity 3783->42518 state reply ifindex lxcdf014: 10.108.27.179:3000 -> 192.168.1.171:59716 tcp RST
<- endpoint 26549 flow 0xb7550b3b identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.205:52436 tcp RST
-> endpoint 49212 flow 0xb7550b3b identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:52436 tcp RST
<- endpoint 65430 flow 0x86726688 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.168:43166 tcp RST
-> endpoint 28921 flow 0x86726688 identity 3783->42518 state reply ifindex lxcca58a: 10.108.27.179:3000 -> 192.168.1.168:43166 tcp RST
<- endpoint 25104 flow 0x949e9218 identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.138:50006 tcp RST
-> endpoint 62268 flow 0x949e9218 identity 3783->42518 state reply ifindex lxc28d65: 10.108.27.179:3000 -> 192.168.1.138:50006 tcp RST
<- endpoint 5751 flow 0x6e719091 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.138:49990 tcp RST
-> endpoint 62268 flow 0x6e719091 identity 3783->42518 state reply ifindex lxc28d65: 10.108.27.179:3000 -> 192.168.1.138:49990 tcp RST
<- endpoint 33229 flow 0xf090e68 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.225:48032 tcp RST
-> endpoint 48734 flow 0xf090e68 identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:48032 tcp RST
<- endpoint 40784 flow 0x795cde5d identity 3783->0 state new ifindex 0: 192.168.1.229:3000 -> 192.168.1.236:59334 tcp RST
-> endpoint 4993 flow 0x795cde5d identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:59334 tcp RST
<- endpoint 5751 flow 0x40ad52f identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.173:44988 tcp RST
-> endpoint 37521 flow 0x40ad52f identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:44988 tcp RST
<- endpoint 25104 flow 0xeb11602c identity 3783->0 state new ifindex 0: 192.168.1.156:3000 -> 192.168.1.232:45064 tcp RST
-> endpoint 654 flow 0xeb11602c identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:45064 tcp RST
<- endpoint 27030 flow 0x7b19317c identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.137:43036 tcp RST
-> endpoint 2350 flow 0x7b19317c identity 3783->42518 state reply ifindex lxcbddb9: 10.108.27.179:3000 -> 192.168.1.137:43036 tcp RST
<- endpoint 65430 flow 0x51534592 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.232:44914 tcp RST
-> endpoint 654 flow 0x51534592 identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:44914 tcp RST
<- endpoint 27030 flow 0x983ff77b identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.165:54856 tcp RST
-> endpoint 37905 flow 0x983ff77b identity 3783->42518 state reply ifindex lxc864e6: 10.108.27.179:3000 -> 192.168.1.165:54856 tcp RST
<- endpoint 26549 flow 0xa86f0c09 identity 3783->0 state new ifindex 0: 192.168.1.179:3000 -> 192.168.1.236:59348 tcp RST
-> endpoint 4993 flow 0xa86f0c09 identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:59348 tcp RST
<- endpoint 40784 flow 0xdbbcb071 identity 3783->0 state new ifindex 0: 192.168.1.229:3000 -> 192.168.1.173:44868 tcp RST
-> endpoint 37521 flow 0xdbbcb071 identity 3783->42518 state reply ifindex lxcef371: 10.108.27.179:3000 -> 192.168.1.173:44868 tcp RST
<- endpoint 27030 flow 0x5118a1c1 identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.205:51318 tcp RST
-> endpoint 49212 flow 0x5118a1c1 identity 3783->42518 state reply ifindex lxc0d49f: 10.108.27.179:3000 -> 192.168.1.205:51318 tcp RST
<- endpoint 33229 flow 0xb8514ffd identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.232:46306 tcp RST
-> endpoint 654 flow 0xb8514ffd identity 3783->42518 state reply ifindex lxcd9432: 10.108.27.179:3000 -> 192.168.1.232:46306 tcp RST
<- endpoint 33229 flow 0x2def67b7 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.165:55902 tcp RST
-> endpoint 37905 flow 0x2def67b7 identity 3783->42518 state reply ifindex lxc864e6: 10.108.27.179:3000 -> 192.168.1.165:55902 tcp RST
<- endpoint 5751 flow 0x8af647d1 identity 3783->0 state new ifindex 0: 192.168.1.154:3000 -> 192.168.1.236:58342 tcp RST
-> endpoint 4993 flow 0x8af647d1 identity 3783->42518 state reply ifindex lxcab2c6: 10.108.27.179:3000 -> 192.168.1.236:58342 tcp RST
<- endpoint 65430 flow 0x4a679f58 identity 3783->0 state new ifindex 0: 192.168.1.181:3000 -> 192.168.1.193:60496 tcp RST
-> endpoint 11557 flow 0x4a679f58 identity 3783->42518 state reply ifindex lxc67dfb: 10.108.27.179:3000 -> 192.168.1.193:60496 tcp RST
<- endpoint 33229 flow 0x8b703872 identity 3783->0 state new ifindex 0: 192.168.1.191:3000 -> 192.168.1.248:35434 tcp RST
-> endpoint 59907 flow 0x8b703872 identity 3783->42518 state reply ifindex lxcd057f: 10.108.27.179:3000 -> 192.168.1.248:35434 tcp RST
<- endpoint 27030 flow 0x34cdae5f identity 3783->0 state new ifindex 0: 192.168.1.240:3000 -> 192.168.1.225:46854 tcp RST
-> endpoint 48734 flow 0x34cdae5f identity 3783->42518 state reply ifindex lxcf8c12: 10.108.27.179:3000 -> 192.168.1.225:46854 tcp RST

whole cilium monitor command log https://gist.github.com/leoh0/46e00a0e1b2e3249a8bcba0c6b8d9776

I do not know which interface to catch by what options, so I took it as a whole, but I did not see the RST packet. whole tcpdump -ennvv log inside cilium-agent https://gist.github.com/leoh0/3bd88d3cac7d4e71d19dc3d1e549a487

Additional resources include:

cilium-agent log https://gist.github.com/leoh0/14e1a2a580fbe55b86cd300110d61d2c

cilium debuginfo command output https://gist.github.com/leoh0/d440fd0c5911674dc43d424cf2a8857b

k8s service, deployment, endpoint yaml https://gist.github.com/leoh0/27221081f4d9ca6fef0bda3750e0c7ae

cilium-agent daemonset yaml https://gist.github.com/leoh0/d8e57bb4f967bccade8ab90b35925220

+2
leoh0 on Sep 7, 2018

@tgraf Great! I checked all the existing problems for a few hours and confirmed that cilium/cilium-dev:v1.2.2-rc2 had no problems at all.

+1
leoh0 on Sep 8, 2018
Read more comments on GitHub
← cilium: RE: Cilium v1.12.2 API issues [Errors encountered while deleting endpoint], [unable to create endpoint, putEndpointIdTooManyRequests]
cilium: Scale-down event of EKS kube-apiserver causes network outage →