kubernetes: event recorder fails to record repeating events

What happened: I created a pod and then wrote a simple go routine to emit an Available event each second. I then watch the events, and see that the LAST SEEN column after a while start to grow, meaning the events stop being received/sent. attached a bash script for easy recreation.

What you expected to happen: since we’re sending an event every second, the LAST SEEN column should show the event with a value of 0/1s.

In actuality after about 0.5min, the events LAST SEEN starts growing (see last line):

Every 2.0s: kubectl get events                               hades01.lab.eng.tlv2.redhat.com: Thu Dec 17 14:41:11 2020

LAST SEEN   TYPE      REASON             OBJECT     MESSAGE
2m42s       Warning   DNSConfigForming   pod/test   Search Line limits were exceeded, some search paths have been omit
ted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local lab.eng.tlv2.redhat.com mgm
t.lab.eng.tlv2.redhat.com eng.lab.tlv.redhat.com
105s        Normal    Scheduled          pod/test   Successfully assigned default/test to kind-control-plane
25s         Warning   DNSConfigForming   pod/test   Search Line limits were exceeded, some search paths have been omit
ted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local lab.eng.tlv2.redhat.com mgm
t.lab.eng.tlv2.redhat.com eng.lab.tlv.redhat.com
104s        Normal    Pulling            pod/test   Pulling image "busybox"
101s        Normal    Pulled             pod/test   Successfully pulled image "busybox" in 2.771330479s
101s        Normal    Created            pod/test   Created container test
101s        Normal    Started            pod/test   Started container test
63s         Normal    Available          pod/test   All applied components are available

How to reproduce it (as minimally and precisely as possible):

1. create a kind cluster (quick kind install)

kind create cluster

1. download, unzip and run the reproducing script. It’ll do the following:

• copy the go routine to the local directory
• create a simple pod “test” on default namespace
• run the go-routine that will record an “Available” event every 1 second

unzip test_script.zip
chmod +x test.sh
./test.sh

1. open another terminal, and watch the events

watch kubectl get events

Anything else we need to know?: I chose showing this with the kubectl get events tool for simplicity, but it also doesn’t work when I tried watching the pod object via k8s api like watcher, err := k8sClient.CoreV1().Pods(namespace).Watch(ctx, metav1.ListOptions{})

Environment:

• Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-14T07:30:52Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):

$ cat /etc/os-release
NAME=Fedora
VERSION="31 (Thirty One)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Thirty One)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"

• Kernel (e.g. uname -a):

$ uname -a
Linux hades01.lab.eng.tlv2.redhat.com 5.8.15-101.fc31.x86_64 #1 SMP Thu Oct 15 16:57:45 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

test_script.zip