kube-mgmt: kube-mgmt throwing errors when pod has been running for quite some time

Unsure if this is an actual issue or if I should log separate issues for each of them.

I am seeing a weird behavior in kube-mgmt after it has ran for a while; eg: more than 5 hours

This logs start appearing

E0904 12:18:45.089926 1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind “Status” is registered for version “v1” in scheme “github.com/open-policy-agent/kube-mgmt/pkg/configmap/configmap.go:102”

kube-mgmt seems to stop honoring the --require-policy-label as all the configmaps in the opa namespace get annotated with the penpolicyagent.org/policy-status: annotation
Resource syncs seem to crash periodically (on an hourly basis)

time=“2019-09-04T10:51:03Z” level=info msg=“Sync channel for extensions/v1beta1/ingresses closed. Restarting immediately.” time=“2019-09-04T10:51:03Z” level=info msg=“Syncing extensions/v1beta1/ingresses.” time=“2019-09-04T10:51:08Z” level=info msg=“Listed extensions/v1beta1/ingresses and got 319 resources with resourceVersion 1188097528. Took 5.462164663s.” time=“2019-09-04T10:52:05Z” level=info msg=“Loaded extensions/v1beta1/ingresses resources into OPA. Took 56.650552306s. Starting watch at resourceVersion 1188097528.”

time=“2019-09-04T11:56:18Z” level=info msg=“Sync channel for extensions/v1beta1/ingresses closed. Restarting immediately.” time=“2019-09-04T11:56:18Z” level=info msg=“Syncing extensions/v1beta1/ingresses.” time=“2019-09-04T11:56:19Z” level=info msg=“Listed extensions/v1beta1/ingresses and got 319 resources with resourceVersion 1188317650. Took 872.624828ms.” time=“2019-09-04T11:57:43Z” level=info msg=“Loaded extensions/v1beta1/ingresses resources into OPA. Took 1m24.203876182s. Starting watch at resourceVersion 1188317650.”

time=“2019-09-04T12:39:50Z” level=info msg=“Sync channel for extensions/v1beta1/ingresses closed. Restarting immediately.” time=“2019-09-04T12:39:50Z” level=info msg=“Syncing extensions/v1beta1/ingresses.” time=“2019-09-04T12:39:53Z” level=info msg=“Listed extensions/v1beta1/ingresses and got 319 resources with resourceVersion 1188471126. Took 2.790341522s.” time=“2019-09-04T12:41:36Z” level=info msg=“Loaded extensions/v1beta1/ingresses resources into OPA. Took 1m43.350371162s. Starting watch at resourceVersion 1188471126.”

Unsure what the root cause may be. But here are some details about the versions being used

kube-mgmt version: 0.9 OPA version: 0.12.2

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 18 (11 by maintainers)

Commits related to this issue

Fixes well known bug: https://github.com/open-policy-agent/kube-mgmt/issues/49 https://github.com/open-policy-agent/kube-mgmt/issues/49 — committed to devops-grundfos/charts by devops-grundfos 4 years ago

Most upvoted comments

We’ve confirmed that v0.11 fixes the logging issue. Since the logging problem and the resource sync issue have been resolved, I’m going to close this for now.

tsandall on Jan 29, 2020

@muzcategui1106-gs great. There’s a new openpolicyagent/kube-mgmt:0.11 image on Docker Hub that you can try out. Please let me know if this resolves the issue.

On Thu, Jan 16, 2020 at 9:00 AM muzcategui1106-gs notifications@github.com wrote:

@tsandall https://github.com/tsandall yes when the release comes out, I will test out the changes in our dev environment

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/open-policy-agent/kube-mgmt/issues/49?email_source=notifications&email_token=AAB2KJP3BTHEESWSFCIG7JDQ6BSBFA5CNFSM4ITROD52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJEE5OA#issuecomment-575164088, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB2KJKDFUUVYGJO32UZ6NLQ6BSBFANCNFSM4ITROD5Q .

– -Torin

tsandall on Jan 16, 2020