oauthenticator: [GitHub] OAuth fails with second computer attempting to log in

Bug description

When I log into JupyterHub with GH OAuth on one machine and then try and log in on a different machine with the same user I get a 500.

Expected behaviour

I would expect to be able to log in

Actual behaviour

Error 500

How to reproduce

Your personal set up

zero-tojupyter-hub, with oauthenticator==0.13.0

Logs

LyJ9 (10.42.0.112)
    HTTPServerRequest(protocol='http', host='holdenkarau.mooo.com', method='GET', uri='/hub/oauth_callback?code=[CODE]&state=[STATE]', version='HTTP/1.1', remote_ip='10.42.0.112')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/oauth2.py", line 224, in get
        user = await self.login_user()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 749, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 462, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/oauthenticator/github.py", line 171, in authenticate
        resp = await http_client.fetch(req)
    tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
    
[E 2021-03-17 19:37:44.650 JupyterHub log:173] {
      "X-Real-Ip": "10.42.0.112",
      "X-Forwarded-Server": "traefik-758cd5fc85-dx5zk",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "holdenkarau.mooo.com",
      "X-Forwarded-For": "10.42.0.112,::ffff:10.42.0.113",
      "Upgrade-Insecure-Requests": "1",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Site": "cross-site",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Dest": "document",
      "Referer": "https://holdenkarau.mooo.com/",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36",
      "Host": "holdenkarau.mooo.com",
      "Connection": "close"
    }
[E 2021-03-17 19:37:44.650 JupyterHub log:181] 500 GET /hub/oauth_callback?code=[secret]&state=[secret] (@10.42.0.112) 469.79ms

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (11 by maintainers)

Most upvoted comments

@minrk Thank you! yeah the scope is configured to read:user. I could 100% see it being some API rate limiting maybe (I run PR dashboard which makes some requests to the GitHub API on a queue). Let me see if I can make that PR dashboard refresh less frequently and maybe the issue will go away 😃

holdenk on Mar 25, 2021

#415 hasn’t been released yet, but should improve error logs by default because the error message from GitHub associated with the 403 will be logged. I’m not sure if GitHub’s error message will be informative, but it’s a start.

The request that’s failing is the equivalent of:

curl -H "Authorization: token $GITHUB_ACCESS_TOKEN" https://api.github.com/user

where $GITHUB_ACCESS_TOKEN is the token retrieved from the oauth response here. If you dump the token itself with a debug statement (print debugging ftw), you can test other requests with it and maybe see what’s going on.

Interestingly, this stage is after a successful completion of the oauth process, so the token for some reason is successfully issued, but is not allowed to access its owner’s model. Maybe a rate limiting event occurred or something? Or something in the scopes?

minrk on Mar 25, 2021

I’m going to try and put some debugging in but I am physically moving my K8s cluster this weekend so might take a bit of time.

holdenk on Mar 21, 2021

Ok I tried it again with:

hub:
# GitHub OAuth doesn't work well with multiple computers and same account :/
  config:
    GitHubOAuthenticator:
      client_id: ID
      client_secret: SECRET
      oauth_callback_url: "https://holdenkarau.mooo.com/hub/oauth_callback"
#      allowed_organizations:
#        - scalingpythonml
      scope:
        - read:user
    Authenticator:
      admin_users:
        - holdenk
      enable_auth_state: true
    CryptKeeper:
      keys:
        - KEY
    DummyAuthenticator:
      password: OTHERBACKUP
#    JupyterHub:
#      authenticator_class: dummy
    JupyterHub:
      authenticator_class: github

holdenk on Mar 19, 2021

That could make sense. I was looking at the configuration parameters and I could only see a way to force it to HTTP (which I don’t want), I’ll take another look this weekend and see (and I’ll share my config tomorrow - the secrets).

holdenk on Mar 18, 2021

Makes sense. I’ll do this in an incognito window tomorrow to verify 😃 Otherwise I’ll see if maybe there’s a cookie to clean I could add to the error message or similar.

holdenk on Mar 17, 2021