kubernetes: GCE IPv4: martian source errors and Node becomes inoperable

Hi all

I am running 1.3 beta, and have had 2 nodes that started in a state where the kernel was throwing martian source errors, and two nodes went to that state.

Here is a snippet of the kernel logs:

Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.359924] Memory cgroup stats for /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426: cache:100KB rss:204700KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:204792KB inactive_file:8KB active_file:0KB unevictable:0KB
Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.402989] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.413053] [ 4051]     0  4051     1112      190       6        0           994 sh
Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.429117] [ 4057]     0  4057    27225     6949      55        0           994 google-fluentd
Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.441524] [ 4062]     0  4062   274400    48880     182        0           994 google-fluentd
Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.451015] Memory cgroup out of memory: Kill process 4062 (google-fluentd) score 1948 or sacrifice child
Jul  4 20:25:49 kubernetes-minion-group-n0xa kernel: [1019057.463970] Killed process 4062 (google-fluentd) total-vm:1097600kB, anon-rss:187684kB, file-rss:7836kB
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.207417] google-fluentd invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=994
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.216157] google-fluentd cpuset=da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426 mems_allowed=0
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.227357] CPU: 5 PID: 28307 Comm: google-fluentd Tainted: G         C    3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707] Hardware name: Google Google, BIOS Google 01/01/2011
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  0000000000000000 ffffffff8150e835 ffff880798508960 ffff8807978efc00
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  ffffffff8150c431 ffff880797f1f060 ffffffff810a7807 0000000000000001
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  ffff880797f1f040 ffff880798b01500 0000000000000001 0000000000000000
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707] Call Trace:
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8150c431>] ? dump_header+0x76/0x1e8
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff810a7807>] ? __wake_up_common+0x57/0x90
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8114073d>] ? find_lock_task_mm+0x3d/0x90
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81140b7d>] ? oom_kill_process+0x21d/0x370
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8114073d>] ? find_lock_task_mm+0x3d/0x90
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff811a087a>] ? mem_cgroup_oom_synchronize+0x52a/0x590
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff8119fe00>] ? mem_cgroup_try_charge_mm+0xa0/0xa0
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81141330>] ? pagefault_out_of_memory+0x10/0x80
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81057505>] ? __do_page_fault+0x3c5/0x4f0
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81171b5f>] ? mprotect_fixup+0x14f/0x270
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.229707]  [<ffffffff81516a28>] ? page_fault+0x28/0x30
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.356846] Task in /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426 killed as a result of limit of /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.381018] memory: usage 204800kB, limit 204800kB, failcnt 231
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.387462] memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.396160] kmem: usage 0kB, limit 18014398509481983kB, failcnt 0
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.404259] Memory cgroup stats for /da9dc66a207af11125ea42262f4c0061cfc126e19e679c39b114858483d37426: cache:100KB rss:204700KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:0KB active_anon:204792KB inactive_file:8KB active_file:0KB unevictable:0KB
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.438068] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.446401] [ 4051j     0  4051     1112      190       6        0           994 sh
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.455984] [ 4057]     0  4057    27225     6949      55        0           994 google-fluentd
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.466727] [28303]     0 28303   289117    48527     171        0           994 google-fluentd
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.477300] Memory cgroup out of memory: Kill process 28303 (google-fluentd) score 1941 or sacrifice child
Jul  5 05:49:31 kubernetes-minion-group-n0xa kernel: [1052879.488688] Killed process 28303 (google-fluentd) total-vm:1156468kB, anon-rss:186272kB, file-rss:7836kB
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.199744] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.207097] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.216441] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.225091] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.244086] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.252793] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.448076] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.455732] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00        r...n..X..1...

The cluster is over 1k in nodes, and is running 1k in Pet Set Cassandra Pets.

Linux info

$ uname -a
Linux kubernetes-minion-group-n0xa 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

Debian version

$ cat /etc/debian_version
7.11

What other info is required?

cc: @bgrant0607 @thockin @Random-Liu

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 3
  • Comments: 33 (30 by maintainers)

Most upvoted comments

Thanks for providing all the information.

+3
freehan on Jul 12, 2016

@freehan so close this??

+3
chrislovecnm on Jul 12, 2016

Okay. I believe you meant beta 2. Beta 2 does not contain the fix. It should not happen now with 1.3 release.

+3
freehan on Jul 12, 2016

Those IP address belong to two different Pets / Pods.

cassandra-data-147                                   1/1       Running             0          15d       10.245.135.3    kubernetes-minion-group-kvcs
cassandra-data-364                                   1/1       Running             0          15d       10.245.49.3     kubernetes-minion-group-n0xa

Here is one of the nodes

$ kubectl describe node kubernetes-minion-group-n0xa
Name:           kubernetes-minion-group-n0xa
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/instance-type=n1-standard-8
            beta.kubernetes.io/os=linux
            failure-domain.beta.kubernetes.io/region=us-central1
            failure-domain.beta.kubernetes.io/zone=us-central1-b
            kubernetes.io/hostname=kubernetes-minion-group-n0xa

The other

$ kubectl describe node kubernetes-minion-group-n0xa
Name:           kubernetes-minion-group-n0xa
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/instance-type=n1-standard-8
            beta.kubernetes.io/os=linux
            failure-domain.beta.kubernetes.io/region=us-central1
            failure-domain.beta.kubernetes.io/zone=us-central1-b

And at least one of the nodes that is in this state is in us-central-1-b. This install is across 4 zones.

+3
chrislovecnm on Jul 8, 2016

We need to know if these martian ips were service Cluster IPs and which one was the Pod CIDR range for this node.

IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0 looks like 2 different podCIDRs. I’ve seen such message when I do something like: iptables -t mangle -A PREROUTING -s pod-ip -j TEE --gateway pod-ip

which will clone packets and send them back to the pod, causing the bridge to think they’re martian. This may happen because of the good old promiscuous bridge, but I haven’t been able to dig up evidence in previous instances.

+3
bprashanth on Jul 8, 2016

In previous cases of fluentd OOM kills, was the node healthy afterwards ?

The previous case happened in a soaking test. We don’t know about the node healthy condition. At least, in https://github.com/kubernetes/kubernetes/issues/27630#issuecomment-226829955, fluentd is oom killed twice during 10 minutes. And the kernel didn’t throw out “martian ip” error during the 10 minutes.

+3
Random-Liu on Jul 8, 2016

Forgot to mention restart of the node fixes this problem.

+3
chrislovecnm on Jul 8, 2016

@girishkalele I need to destroy the entire cluster, and how do I get the info you need.

+3
chrislovecnm on Jul 8, 2016

Previous cases?

+3
chrislovecnm on Jul 8, 2016 
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.199744] IPv4: martian source 10.245.135.3 from 10.245.49.3, on dev cbr0
Jul  6 20:54:02 kubernetes-minion-group-n0xa kernel: [1193550.207097] ll header: 00000000: 72 c3 ed 18 6e 01 0a 58 0a f5 31 03 08 00

10.245.135.3 and 10.245.49.3 belongs to what?

+2
freehan on Jul 8, 2016
Read more comments on GitHub
← kubernetes: GCE Cloud: Service (LoadBalancer) networking breaks when pod is restarted
kubernetes: gce-ipvs CI job failures →