kubernetes: Huge amount of logs: kubelet_getters.go:300] "Path does not exist" path="/var/lib/kubelet/pods/86cba354-348c-4826-9837-df9c616a8862/volumes"

I stumbled upon this issue, as we have the same problem on all our Rancher RKE (k8s v1.24 and v1.25) clusters where the kubelet container is spamming "Path does not exist" messages - this resulted in gigabytes of logs from one of our clusters which is running dozens of kubernetes-jobs on ~3 nodes.

For a quick and ugly fix I created the script below that periodically cleans the corresponding /sys/fs/cgroup/misc-folders via a simple cronjob (* * * * * root /opt/scripts/cgroup_pod_garbage_collector.sh -i 50s >/dev/null 2>&1) - not sure if it’s super smart what I’m doing and the script can surely be done in a much better way, but we needed a quick fix, the kubelet containers on the nodes caused extremely high CPU-usage writing those logs and our ELK also complained about space-usage 😉

#!/bin/bash

##################################################### ARGUMENTS - BEGIN
usage() {
   echo "Script for cleaning up cgroup garbage of pods that got deleted" 1>&2
   echo " " 1>&2
   echo "Usage: $0 [-i <interval>]    example: set to cronjob-interval but slightly lower to not get overlaps for example" 1>&2
   echo "                                                                   if it runs every 5 minutes, set 290s" 1>&2
   exit 1
}

while getopts ":i:" o; do
    case "${o}" in
        i)
            RUN_INTERVAL=${OPTARG}
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

if [ -z "${RUN_INTERVAL}" ]; then
    usage
fi
##################################################### ARGUMENTS - END

##################################################### CHECK IF ALREADY RUNNING - BEGIN
# From https://gist.github.com/beugley/43f3feb8c532f7cc07cbcb3ea3ccd203 - slightly adapted
#
# To prevent multiple instances of a script from executing concurrently,
# include this function, and a statement to invoke it, at the top of any
# ksh/bash script.  This function will check for other instances of the
# same script that are executing by the same user.  If any are found, then
# this instance terminates immediately.
#
# When scripts are invoked in a repetitive manner (cron or other scheduler),
# it's possible that an instance can still be running when the time arrives
# to start a new instance.  For example, suppose that you schedule a script
# to run every 15 minutes, but one run takes 20 minutes?  You may not want
# 2 (or more) concurrent instances of the script to run.
#
terminateIfAlreadyRunning() {
    ##
    ## Terminate if another instance of this script is already running.
    ##
    SCRIPT_NAME=`/bin/basename $1`
    echo "`/bin/date +%Y%m%d.%H%M%S:` This instance of $SCRIPT_NAME is PID $$"
    for PID in `/bin/ps -fC $SCRIPT_NAME | /bin/grep "^$USER" | /usr/bin/awk '{print $2}'`; do
        ALIVE=`/bin/ps -p $PID -o pid=`
        if [[ "$PID" != "$$"  &&  "$ALIVE" != "" ]]
        then
                echo "WARNING: $SCRIPT_NAME is already running as process $PID!"
                echo "Terminating..."
                exit 0
        fi
    done
}

terminateIfAlreadyRunning $0
##################################################### CHECK IF ALREADY RUNNING - END

##################################################### FUNCTIONS - BEGIN
# blatantly stolen from https://www.urbanautomaton.com/blog/2014/09/09/redirecting-bash-script-output-to-syslog/
# for nicely logging to console and /var/log/messages
readonly SCRIPT_NAME=$(basename $0)

log_info() {
  echo "$@"
  logger -p local0.info -t $SCRIPT_NAME "$@"
}

log_warn() {
  echo "$@"
  logger -p local0.warning -t $SCRIPT_NAME "$@"
}

log_err() {
  echo "$@" >&2
  logger -p local0.err -t $SCRIPT_NAME "$@"
}

# testing
#log_info "writing to stdout"
#log_err "writing to stderr"
##################################################### FUNCTIONS - END

##################################################### MAIN-SCRIPT - BEGIN
# we get the "IDs" of the pods
KUBE_POD_IDS=$(docker logs kubelet --since ${RUN_INTERVAL} 2>&1 | grep "Path does not exist" | awk '{ print $9 }' | awk -F "/" '{print $6}' | sort | uniq)

for POD_ID in $KUBE_POD_IDS; do
    # we search for the folder below sys - as observed it either can be in .../besteffort/... or .../burstable/...
    # the resulting path should look something like "/sys/fs/cgroup/misc/kubepods/besteffort/pod${POD_ID}"
    # TODO: can there be multiple paths? we may have to handle that!
    CGROUP_DIR_NAME=$(find /sys -name "*${POD_ID}")
    # was something found?
    if [ -z "$CGROUP_DIR_NAME" ]; then
        log_warn "nothing found in /sys for ${POD_ID}!"
    else
        if [ -d "${CGROUP_DIR_NAME}" ]; then
            # "rm -rf" not working with cgroup's pseudo-filesystem
            if rmdir "${CGROUP_DIR_NAME}"; then
                log_info "successfully deleted ${CGROUP_DIR_NAME}"
            else
                log_err "could not delete ${CGROUP_DIR_NAME}!"
            fi
        else
            log_warn "already removed ${CGROUP_DIR_NAME} earlier!"
        fi
    fi
done
##################################################### MAIN-SCRIPT - END

I haven’t seen any issues yet on the clusters where this script is active and docker logs for the kubelet container also look much much cleaner. Log-space in our ELK was reduced from for example ~15G per hour from one of the nodes to ~3G per hour. CPU usage also dropped significantly from 20% to 4% after executing the script manually on one of the nodes.

Looks like it’s a problem with runc <= 1.1.3

After upgrading runc everyhting works again.

edit: nope, loglines are still present with k8s v1.24.4 and runc 1.1.4. But the memory leak and cgroup error messages seem to be gone.

edit: typo in runc versions. 1.1.3 and 1.1.4, not 1.3/1.4.

Hi,

We are observing the same issue on Bottlerocket official AMI images. Suddenly a node during the night gets flooded with those errors and fills up the disk.

Does anyone run Bottlerocket OS ?

In my case the same ids kept showing up for hours. This was due to the misc cgroup not being properly cleaned up. And because not everything was cleaned up the pods kept getting picked up by the housekeeping loop.

On the first run the directory is removed. all consecutive runs this log is produced.

So while this log isn’t the problem it is (the only) indication that the kubelet fails to completely delete pods from the system.