jx: Hook and Tide pods failing with credentials error
Summary
Both Hook and Tide pods have started failing into CrashLoopBackOff with the same “Bad credentials” error against GitHub.
{"component":"tide","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-03-21T10:15:31Z"}
Steps to reproduce the behavior
Unclear. This server has been running for about three days on preemptible nodes. The fault appears to have started a little while after upgrading the platform install and subsequent to a node being automatically flushed.
Expected behavior
Expected that Hook and Tide would continue to be able to connect to GitHub.
Actual behavior
Both pods seem to have lost the credentials needed to connect to GitHub. Are they unable to persist their config through a restart?
Jx version
The output of jx version
is:
NAME VERSION
jx 1.3.998
jenkins x platform 0.0.3588
Kubernetes cluster v1.11.7-gke.4
kubectl v1.13.4
helm client Client: v2.13.0+g79d0794
git git version 2.21.0
Operating System Mac OS X 10.13.6 build 17G4015
Jenkins type
- Classic Jenkins
- Serverless Jenkins
Kubernetes cluster
jx create cluster gke \
--cluster-name='d27' \
--default-admin-password='xxxxx' \
--environment-git-owner='tdcox' \
--enhanced-apis=true \
--enhanced-scopes=true \
--git-username='tdcox' \
--git-private=false \
--kaniko=true \
--labels='demo=true' \
--machine-type='n1-standard-4' \
--max-num-nodes='3' \
--min-num-nodes='2' \
--no-tiller=true \
--preemptible=true \
--project-id='jx-mar19' \
--prow=true \
--skip-login=true \
--tekton=true \
--vault=false \
--zone='europe-west1-d'
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 29 (2 by maintainers)
I have the same issues when trying to install JX Serverless on Github Enterprise. Installation goes through but Hook and Tide pods are failing to run due to bad credentials (using Open Github instead). Tried even to remove the GitHub git server but still using that one. Then, it seems the serverless approach works only with Github (not GHE), so I see Static Jenkins as the only option 😕 (I hope I am wrong). Anyway, I am wondering if there is any workaround or if you plan to have a look at this issue.
@ccojocar I am observing the same error in a serverless (tekton + prow) JenkinsX installation on EKS. I was unsure whether it made sense to create a new issue so I decided to add my observations / questions here. Apologies in advance if this turns to be not appropriate.
My installation consists of an on-premise BitBucket server so I find it strange that
hook
pods complain about GitHub credentials. Is it possible to configure a serverless Jenkins installation without a GitHub account?Same issue, fresh install, no vault…
Hi, it looks like I got bitten by this too.
Setup:
The hook and tide deployments failed and their current status is “CrashLoopBackOff”. Checking their logs I see errors related to github bad credentials, even though I’m trying to use bitbucket…
The various logs show:
$ kubectl logs tide-548574ff88-tcnhk
{“client”:“github”,“component”:“tide”,“controller”:“sync”,“level”:“info”,“msg”:“Throttle(800, 39)”,“time”:“2019-07-25T13:33:15Z”} {“client”:“github”,“component”:“tide”,“controller”:“status-update”,“level”:“info”,“msg”:“Throttle(400, 200)”,“time”:“2019-07-25T13:33:15Z”} {“client”:“github”,“component”:“tide”,“level”:“info”,“msg”:“User()”,“time”:“2019-07-25T13:33:15Z”} {“component”:“tide”,“error”:"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3\“}”,“level”:“fatal”,“msg”:“Error getting Git client.”,“time”:“2019-07-25T13:33:16Z”} $ kubectl logs hook-954dbdf4d-btbxg
{“client”:“github”,“component”:“hook”,“level”:“info”,“msg”:“User()”,“time”:“2019-07-25T13:33:21Z”} {“component”:“hook”,“error”:"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3\“}”,“level”:“fatal”,“msg”:“Error getting Git client.”,“time”:“2019-07-25T13:33:22Z”} $ kubectl logs hook-954dbdf4d-dvpjh
{“client”:“github”,“component”:“hook”,“level”:“info”,“msg”:“User()”,“time”:“2019-07-25T13:33:22Z”} {“component”:“hook”,“error”:"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3\“}”,“level”:“fatal”,“msg”:“Error getting Git client.”,“time”:“2019-07-25T13:33:22Z”}
This is a test cluster so I’m happy to provide more details and/or try things which might potentially break it. Just let me know if I can help further please.
Thanks
@vfarcic it also prevents people from installing new clusters, so I’m hoping this issue is actually addressed…