weave: Memory leak/OOM with "Received update for IP range I own" messages in log

OK so since the weave nodes became consistent, things have been stable with no memory growth or OOM issues. I have done a minor amount of scaling (perhaps up/down of 10 nodes or so) and things have remained consistent throughout that with 2.5.2.

I wrote a quick script verify-weave.sh which will go through all weave pods in the cluster, compute a checksum on the ‘status ipam’ list and tell you if there are any weave pods that disagree on their peer list. We used this to identify the different groups of pods in the cluster and decide which we wanted to preserve and which ones we wanted to reset/restart. Another quick script bump-weave.sh was then used to remove the db file and restart for those weave pods we wished to reset.

Both of those scripts were written quickly to address a particular condition here and so are not intended as portable or good examples of coding, but may be useful to someone so here they are.

@itskingori , @murali-reddy - thank you both for your help on this.

@itskingori - I will keep an eye on it for sure. We’re not doing a huge amount of scaling at the moment, however we did just do some refactoring of instance types hence we had a fairly large amount of hosts created and deleted which is what set the current condition into motion. There is some more similar adjustments still outstanding so I will check and let you know how things look as that happens.

Also, thanks very much for your feedback and detail - sounds like your process to recover normal operation was very similar to mine. The only difference seems to be that you rebooted/terminated the nodes whereas I just recycled the weave pod on the node and left the other pods and the node itself alone. We’ll be doing this same process in our prd cluster today so if I cook up a noteworthy script I will share it here. Cheers -

Yes, there are no ramifications. For now these manual steps will reconcile from state where there were IPAM conflicts. In 2.6 release IPAM conflicts are automatically resolved (https://github.com/weaveworks/weave/pull/3637)

@itskingori - interesting. I ran the following on my prd cluster, which is having this symptom, and I also see there are two distinct set of “status ipam” hosts returned from the cluster, each comprising about 50% of the nodes:

# Generate a sorted list of hosts-only from ipam output per weave node
$ for s in `kubectl get pod | grep weave-net | awk '{ print $1 }'`; do kubectl exec -ti $s -c weave -- ./weave --local status ipam | awk '{ print $1 }' | sort > $s.ipam; done

# See how many different variants of that list there are - this shows there are two different checksums for the files, and there are 38 of one and 33 of the other.
$ sum *.ipam | awk '{ print $1 }' | sort | uniq -c
     38 02398
     33 57603

Yes, this is more or less the position we were in. I’ll borrow your commands are they seem similar to (if not better than) what I have to look through weave 👇

#!/bin/bash

set -u

context=$1
command=$2
pods=$(kubectl get pods -n=kube-system -lname=weave-net --context=${context} | tail -n+2 | awk '{print $1}')

for pod in ${pods[@]}
do
  kubectl exec -n=kube-system --context=${environment} ${pod} -- /home/weave/weave --local ${command}
done

I use something like ./script.sh <cluster> "<status ipam>" to loop through all the weave pods.

I wonder if I can do something with that one host that’s in one list but not the other? Or if I do indeed need to restart all the systems in one of the lists…

Weave works by sharing state by consensus, the fact that there are two states breaks weave and you need to bite the bullet and get rid of a group or any weave pods that have inconsistent state. There are two ways to do this:

1. Terminate all the nodes with bad state at the same time.
2. Terminate weave pods with bad state while deleting their database (that stays on the host) at the same time.

I ran with no. 2 because this was production so recovery was critical. I didn’t have time to write a script for no. 2. I pretty much figured out which group to terminate and terminated them all at the same time. I’m guessing, it’s better to terminate weave while deleting the database on the host so that the new pod starts from a clean slate and gets its state from other correct weave pods.

When you say you terminated the hosts in one of the groups, did you do anything to the remaining systems or just terminate the nodes in one of the lists?

I didn’t do anything to the remaining group as their state was similar and correct. Once you get rid of the bad weave pods, everything goes back to normal and the state is now consistent among those remaining … the problem goes away, and the cluster heals.

Also did the nodes that were terminated have anything special done on them prior (such as running weave reset or similar?) .

I didn’t do anything to the ones remaining. The fact that they had similar state was all I needed.

I’m also curious if you did them one at a time, and somehow observe that they came up with a consistent ipam list?

I terminated them all at the same time because I could not risk them sharing state any longer. I figured doing it one by one might not work because new pod might get it state from another bad one. I wanted all the ‘bad’ ones gone and only the ‘good’ ones left to share state.

Basically I think I need to do the same thing here, just want to make sure that deleting those nodes won’t just increase the amount of breakage as it seems like node removal is what precipitated all of this in the first place.

It didn’t cause more breakage for me. Other than the time when I had to wait for the autoscaling group to replace the nodes that I just terminated. And the disruption it caused to the apps on the nodes I just terminated as new pods come up.

Here are the log files from the other two weave pods producing the “Received update for IP range I own” log messages - only 3 of the 71 pods are producing this message (the other is in the original comment)

weave-net-67hrw.log weave-net-cqfc9.log