kubernetes: Kubelet is not creating the log symlink

@joelittlejohn Here’s my solution which is based on yours. This uses a DaemonSet. It replaces symlinks in both /var/log/containers and /var/log/pods, since I was having trouble with that.

log-linker.sh:

#!/usr/bin/env bash

#
# Derived from https://github.com/kubernetes/kubernetes/issues/52172#issuecomment-346075080
#
set -eu
if [ ! -d /var/lib/docker/containers ]; then
    echo "/var/lib/docker/containers doesn't exist; aborting" 1>&2
    exit 1
fi

cd /var/lib/docker/containers

while true ; do
    for DOCKER_ID in *; do
        CONTAINER="$(cat ${DOCKER_ID}/config.v2.json | jq '{Labels:.Config.Labels, LogPath, ID}')"
        CONTAINER_ID="$(echo ${CONTAINER} | jq -r .ID)"
        CONTAINER_NAME="$(echo ${CONTAINER} | jq -r '.Labels["io.kubernetes.container.name"]')"
        LOG_PATH="$(echo ${CONTAINER} | jq -r .LogPath)"
        POD_NAME="$(echo ${CONTAINER} | jq -r '.Labels["io.kubernetes.pod.name"]')"
        POD_NAMESPACE="$(echo ${CONTAINER} | jq -r '.Labels["io.kubernetes.pod.namespace"]')"
        LINK1_NAME="$(printf "%s_%s_%s-%s" "$POD_NAME" "$POD_NAMESPACE" "$CONTAINER_NAME" "$CONTAINER_ID" | cut -c 1-251)"
        LINK1_FILENAME=$(printf "/var/log/containers/%s.log" "$LINK1_NAME")
        LINK2_FILENAME=$(echo ${CONTAINER} | jq -r '.Labels["io.kubernetes.container.logpath"]')

        if [ -n "${LINK1_FILENAME}" -a ! -e "${LINK1_FILENAME}" -a -e "${LOG_PATH}" ] ; then
            echo "Missing log symlink ${LINK1_FILENAME} for ${LOG_PATH}, creating it now"
            ln -s ${LOG_PATH} ${LINK1_FILENAME}
        fi

        if [ "${LINK2_FILENAME}" = "null" ] ; then
            LINK2_FILENAME=""
        fi
        if [ -n "${LINK2_FILENAME}" -a ! -e "${LINK2_FILENAME}" -a -e "${LINK1_FILENAME}" ] ; then
            echo "Missing log symlink ${LINK2_FILENAME} for ${LINK1_FILENAME}, creating it now"
            if [ ! -d "$(dirname ${LINK2_FILENAME})" ] ; then
                mkdir -p "$(dirname ${LINK2_FILENAME})"
            fi
            ln -s ${LINK1_FILENAME} ${LINK2_FILENAME}
        fi
    done

    sleep 60
done

Dockerfile:

FROM debian

RUN apt-get update && \
    apt-get install -y \
        jq

ADD log-linker.sh /

deploy.yaml:

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: log-linker
  namespace: kube-system
  labels:
    app: log-linker
spec:
  selector:
    matchLabels:
      app: log-linker
  template:
    metadata:
      labels:
        app: log-linker
    spec:
      containers:
      - name: log-linker
        image: your/image/path/here:latest
        command:
        - /log-linker.sh
        volumeMounts:
        - name: var-lib-docker
          mountPath: /var/lib/docker
        - name: var-log
          mountPath: /var/log
      volumes:
      - name: var-log
        hostPath:
          path: /var/log
      - name: var-lib-docker
        hostPath:
          path: /var/lib/docker

(This code is under the MIT License.)

Is this being worked on? I think we are hitting this issue.

still reproducible

For those watching this issue, joelittlejohn and cjyar updated their comments to license the code under MIT (you don’t get notifications for edits). Thanks guys!

@joelittlejohn @cjyar Could you guys to attach a license to that code (e.g. MIT) so that others can (safely) use it?

It’s not fancy, but here’s what I came up with.

I add the following units to the CoreOS cloud-config for my nodes:

    - name: ensure-log-symlinks.service
      command: start
      content: |
        [Unit]
        Description=Ensure kubernetes has not deleted log symlinks
        [Service]
        Type=oneshot
        ExecStart=/bin/bash /etc/kubernetes/ensure-log-symlinks.sh
    - name: ensure-log-symlinks.timer
      command: start
      content: |
        [Unit]
        Description=Ensure kubernetes has not deleted log symlinks
        [Timer]
        OnUnitActiveSec=1min
        OnBootSec=10min
        [Install]
        WantedBy=timers.target

then add this script to the write-files section of the cloud-config (note: requires jq):

  - path: /etc/kubernetes/ensure-log-symlinks.sh
    owner: root
    permissions: 0744
    content: |
      #!/usr/bin/bash
      #
      # See https://github.com/kubernetes/kubernetes/issues/52172#issuecomment-328819603
      #
      set -eu
      if [ -d /var/lib/docker/containers ]; then
        pushd /var/lib/docker/containers
        for POD_ID in *; do
          CONTAINER="$(cat $POD_ID/config.v2.json | jq '{Labels:.Config.Labels, LogPath, ID}')"
          CONTAINER_ID="$(echo $CONTAINER | jq -r .ID)"
          CONTAINER_NAME="$(echo $CONTAINER | jq -r '.Labels["io.kubernetes.container.name"]')"
          LOG_PATH="$(echo $CONTAINER | jq -r .LogPath)"
          POD_NAME="$(echo $CONTAINER | jq -r '.Labels["io.kubernetes.pod.name"]')"
          POD_NAMESPACE="$(echo $CONTAINER | jq -r '.Labels["io.kubernetes.pod.namespace"]')"
          SYMLINK_NAME="$(printf "%s_%s_%s-%s" "$POD_NAME" "$POD_NAMESPACE" "$CONTAINER_NAME" "$CONTAINER_ID" | cut -c 1-251)"
          SYMLINK_FILENAME=$(printf "/var/log/containers/%s.log" "$SYMLINK_NAME")

          if [ ! -f $SYMLINK_FILENAME ]; then
            echo "Missing log symlink $SYMLINK_FILENAME for $LOG_PATH, creating it now" | systemd-cat -t ensure-log-symlinks.sh
            ln -s $LOG_PATH $SYMLINK_FILENAME
          fi
        done
        popd
      fi

Of course for anyone that doesn’t want to create a CoreOS (or similar) cloud-config in this way, you can just take the ensure-log-symlinks.sh script content and use that.

Note: All code in this comment is licensed by me, the author, under the MIT License.

I occuer the same problem. After many tries , I found the cause is that I changed docker log driver from json to others. When i change it back, it fixed. I hope that can help u.

Hello joel,

I understand that the effort to upgrade is not simple, so if the change is merge you will need to port of upgrade. I have build a workaround for a client some time ago, still not ideal but it does fix the issue.

Ideally you can run a daemonset that evaluate the sysmlinks and recreated when this symlink is deleted by the kubelet every minute.

You will need access to the docker interface, and the directory.

This pice of code is all you need.

        const ext4MaxFileNameLen = 255
	podFullName := podName + "_" + podNamespace

	suffix := fmt.Sprintf(".%s", "log")
	logPath := fmt.Sprintf("%s_%s-%s", podFullName, containerName, dockerId)
	// Length of a filename cannot exceed 255 characters in ext4 on Linux.
	if len(logPath) > ext4MaxFileNameLen-len(suffix) {
		logPath = logPath[:ext4MaxFileNameLen-len(suffix)]
	}
	logSymLink := path.Join(containerLogsDir, logPath+suffix)

	_, err := os.Stat(logSymLink)
	if err != nil {	
		info, err := cli.InspectContainer(dockerId)
               err := os.Symlink(info.LogPath, logSymLink)
        }
}

Regards.