core: cr-syncer stops working on authorization errors
When cr-syncer faces an Unauthorized error (I assume at Kubernetes API endpoint of the cloud cluster) it stops working completely and repeats this error message multiple times per second.
E1209 15:26:45.385515 1 reflector.go:153] external/io_k8s_client_go/tools/cache/reflector.go:105: Failed to list *unstructured.Unstructured: Unauthorized
In this state cr-syncer is not syncing any CR. After deleting the pod, cr-syncer starts working again.
This error usually occurs when you run the setup_robot.sh script again on a robot which is already setup. Maybe cr-syncer is running on an old (invalid) REST config for the remote Kubernetes API server when this happens. As this error does not occur very often in my opinion it would be ok to simply crash cr-syncer on this error.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (16 by maintainers)
Commits related to this issue
- Restart cr-syncer after setup-robot. This prevents the cr-syncer from continuing with an invalid token, and so works around https://github.com/googlecloudrobotics/core/issues/59. Change-Id: Ic13363d... — committed to drigz/core by drigz 4 years ago
- Add livenessProbe to cr-syncer This will restart the cr-syncer if it fails to list the robots in the remote cluster three times in a row. I hope this will reduce the severity of https://github.com/g... — committed to drigz/core by drigz 3 years ago
- Add livenessProbe to cr-syncer This will restart the cr-syncer if it fails to list the robots in the remote cluster three times in a row. I hope this will reduce the severity of https://github.com/g... — committed to googlecloudrobotics/core by drigz 3 years ago
Good suggestion, thanks! I’ve merged this now, so please keep us updated on whether you see the issue again.