kubernetes: Possible Bug: Garbage Collection Not Triggered

what

I’m struggling to trigger image garbage collection (GC) by the kubelet process by simulating disk usage. In short, I can’t get it to work and can’t figure out if it’s a bug or a misunderstanding.

expectations

• Upon disk usage crossing the 90% threshold (default setting for image-gc-high-threshold) the kubelet will output messages to stdout or stderr indicating that it’s vacuuming images
• A reduction of disk space below the high-watermark of 90%

related

One of our problems was probably as a result of using a symlinked docker-root (see: https://github.com/kubernetes/kubernetes/issues/17994). We’ve since deployed the kubelet with —docker-root=/vol/docker as well as explicitly passed the —graph=/vol/docker directory to the docker daemon. Despite these changes images are not being gc’d.

reproduction

To simulate being out of space, I’ve created a tmpfile that uses up 89% of free space.

dd if=/dev/zero of=/vol/docker/tmpfile bs=1G count=75

(I should have used a smaller volume!)

Disk space is now at 89% utilization as reported by df

Then I’ve downloaded 11G of docker images with docker pull by iterating over every release of an internal project to take me over the line to 91% as reported by df.

From what I observe from journalctl is that the kubelet is not performing any GC and the df does not change. I’ve also tried restarting the kublet to no avail.

kubernetes version

Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}