jx: Hook and Tide pods failing with credentials error

Summary

Both Hook and Tide pods have started failing into CrashLoopBackOff with the same “Bad credentials” error against GitHub.

{"component":"tide","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-03-21T10:15:31Z"}

Steps to reproduce the behavior

Unclear. This server has been running for about three days on preemptible nodes. The fault appears to have started a little while after upgrading the platform install and subsequent to a node being automatically flushed.

Expected behavior

Expected that Hook and Tide would continue to be able to connect to GitHub.

Actual behavior

Both pods seem to have lost the credentials needed to connect to GitHub. Are they unable to persist their config through a restart?

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.998
jenkins x platform 0.0.3588
Kubernetes cluster v1.11.7-gke.4
kubectl            v1.13.4
helm client        Client: v2.13.0+g79d0794
git                git version 2.21.0
Operating System   Mac OS X 10.13.6 build 17G4015

Jenkins type

  • Classic Jenkins
  • Serverless Jenkins

Kubernetes cluster

jx create cluster gke \
--cluster-name='d27' \
--default-admin-password='xxxxx' \
--environment-git-owner='tdcox' \
--enhanced-apis=true \
--enhanced-scopes=true \
--git-username='tdcox' \
--git-private=false \
--kaniko=true \
--labels='demo=true' \
--machine-type='n1-standard-4' \
--max-num-nodes='3' \
--min-num-nodes='2' \
--no-tiller=true \
--preemptible=true \
--project-id='jx-mar19' \
--prow=true \
--skip-login=true \
--tekton=true \
--vault=false \
--zone='europe-west1-d'

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 29 (2 by maintainers)

Most upvoted comments

I have the same issues when trying to install JX Serverless on Github Enterprise. Installation goes through but Hook and Tide pods are failing to run due to bad credentials (using Open Github instead). Tried even to remove the GitHub git server but still using that one. Then, it seems the serverless approach works only with Github (not GHE), so I see Static Jenkins as the only option 😕 (I hope I am wrong). Anyway, I am wondering if there is any workaround or if you plan to have a look at this issue.

@ccojocar I am observing the same error in a serverless (tekton + prow) JenkinsX installation on EKS. I was unsure whether it made sense to create a new issue so I decided to add my observations / questions here. Apologies in advance if this turns to be not appropriate.

{"client":"github","component":"hook","level":"info","msg":"User()","time":"2019-04-23T01:15:50Z"}
{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-04-23T01:15:50Z"}

My installation consists of an on-premise BitBucket server so I find it strange that hook pods complain about GitHub credentials. Is it possible to configure a serverless Jenkins installation without a GitHub account?

Same issue, fresh install, no vault…

{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-06-11T09:32:22Z"}
jx version
NAME               VERSION
jx                 2.0.261
jenkins x platform 2.0.330
Kubernetes cluster v1.13.1
kubectl            v1.14.0
helm client        Client: v2.11.0+g2e55dbe
git                git version 2.21.0

Hi, it looks like I got bitten by this too.

Setup:

  • this is an EKS cluster setup with the eksctl utility, not with jenkins-x
  • then I installed jenkins-x with the following command: jx install --verbose --provider=eks --no-default-environments --git-provider-url “https://bitbucket.org” --git-username “<my_username>” --git-api-token “<my_token>”

The hook and tide deployments failed and their current status is “CrashLoopBackOff”. Checking their logs I see errors related to github bad credentials, even though I’m trying to use bitbucket…

The various logs show:

$ kubectl logs tide-548574ff88-tcnhk
{“client”:“github”,“component”:“tide”,“controller”:“sync”,“level”:“info”,“msg”:“Throttle(800, 39)”,“time”:“2019-07-25T13:33:15Z”} {“client”:“github”,“component”:“tide”,“controller”:“status-update”,“level”:“info”,“msg”:“Throttle(400, 200)”,“time”:“2019-07-25T13:33:15Z”} {“client”:“github”,“component”:“tide”,“level”:“info”,“msg”:“User()”,“time”:“2019-07-25T13:33:15Z”} {“component”:“tide”,“error”:"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3\“}”,“level”:“fatal”,“msg”:“Error getting Git client.”,“time”:“2019-07-25T13:33:16Z”} $ kubectl logs hook-954dbdf4d-btbxg
{“client”:“github”,“component”:“hook”,“level”:“info”,“msg”:“User()”,“time”:“2019-07-25T13:33:21Z”} {“component”:“hook”,“error”:"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3\“}”,“level”:“fatal”,“msg”:“Error getting Git client.”,“time”:“2019-07-25T13:33:22Z”} $ kubectl logs hook-954dbdf4d-dvpjh
{“client”:“github”,“component”:“hook”,“level”:“info”,“msg”:“User()”,“time”:“2019-07-25T13:33:22Z”} {“component”:“hook”,“error”:"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3\“}”,“level”:“fatal”,“msg”:“Error getting Git client.”,“time”:“2019-07-25T13:33:22Z”}

This is a test cluster so I’m happy to provide more details and/or try things which might potentially break it. Just let me know if I can help further please.

Thanks

@vfarcic it also prevents people from installing new clusters, so I’m hoping this issue is actually addressed…