kubeflow: Jupyter - Http 503 - upstream connect error or disconnect/reset before headers

Step by step

Installation following the instructions here: https://www.kubeflow.org/docs/started/getting-started-gke/
Connect to Kubeflow homepage:

Click JupyterHub and start a new Notebook container
The user container is properly created

I am being redirected to https://gde-ai-test-kubeflow.endpoints.gde-core-dev.cloud.goog/hub/api/oauth2/authorize?client_id=jupyterhub-user-accounts.google.com%253Ajerome%2540gigster.com&redirect_uri=%2Fuser%2Faccounts.google.com%253Ajerome%40gigster.com%2Foauth_callback&response_type=code&state=eyJ.. and getting the error upstream connect error or disconnect/reset before headers

When refreshing, different request get a 503:

Note:

Any URl after to /hub or /user gives the same error: HTTP 503
Tried also with RC 3

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 27 (20 by maintainers)

Most upvoted comments

@jlewi I guess this issue is not related with #1721.

Since, the #1721 issue will only affect the jupyter notebook. This issue happened between ambassador and jupyterhub.

I will check it later today.

leoncamel on Oct 10, 2018

Just FYI, expose jupyterhub-0 service can be a walk-around:

kubectl port-forward -n kubeflow jupyterhub-0 8000:8000

leoncamel on Oct 7, 2018

In my configuration, following the documentation for the GKE, it seems to be using Envoy in front of the JupyterHub.

Adding the Envoy log for twice the exact same request:

Not working

[debug][pool] external/envoy/source/common/http/http1/conn_pool.cc:99] queueing request due to no available connections
[trace][http] external/envoy/source/common/http/conn_manager_impl.cc:644] [C358][S13463747193362273593] decode headers called: filter=0x3dafa80 status=1
[trace][http] external/envoy/source/common/http/http1/codec_impl.cc:315] [C358] parsed 1740 bytes
[trace][connection] external/envoy/source/common/network/connection_impl.cc:229] [C358] readDisable: enabled=true disable=true
[trace][connection] external/envoy/source/common/network/connection_impl.cc:377] [C358] socket event: 2
[trace][connection] external/envoy/source/common/network/connection_impl.cc:445] [C358] write ready
[trace][connection] external/envoy/source/common/network/connection_impl.cc:377] [C359] socket event: 3
[trace][connection] external/envoy/source/common/network/connection_impl.cc:445] [C359] write ready
[debug][connection] external/envoy/source/common/network/connection_impl.cc:464] [C359] delayed connection error: 111
[debug][connection] external/envoy/source/common/network/connection_impl.cc:134] [C359] closing socket: 0

working

[trace][http] external/envoy/source/common/http/conn_manager_impl.cc:644] [C352][S9918074447228879897] decode headers called: filter=0x3daf3c0 status=1
[trace][http] external/envoy/source/common/http/http1/codec_impl.cc:315] [C352] parsed 1740 bytes
[trace][connection] external/envoy/source/common/network/connection_impl.cc:229] [C352] readDisable: enabled=true disable=true
[trace][connection] external/envoy/source/common/network/connection_impl.cc:377] [C357] socket event: 2
[trace][connection] external/envoy/source/common/network/connection_impl.cc:445] [C357] write ready
[debug][connection] external/envoy/source/common/network/connection_impl.cc:455] [C357] connected
[debug][client] external/envoy/source/common/http/codec_client.cc:52] [C357] connected
[debug][pool] external/envoy/source/common/http/http1/conn_pool.cc:225] [C357] attaching to next request
[debug][router] external/envoy/source/common/router/router.cc:948] [C352][S9918074447228879897] pool ready
[trace][connection] external/envoy/source/common/network/connection_impl.cc:314] [C357] writing 1794 bytes, end_stream false
[trace][connection] external/envoy/source/common/network/raw_buffer_socket.cc:67] [C357] write returns: 1794
[trace][connection] external/envoy/source/common/network/connection_impl.cc:377] [C352] socket event: 2
[trace][connection] external/envoy/source/common/network/connection_impl.cc:445] [C352] write ready
[trace][connection] external/envoy/source/common/network/connection_impl.cc:377] [C357] socket event: 2
[trace][connection] external/envoy/source/common/network/connection_impl.cc:445] [C357] write ready
[trace][connection] external/envoy/source/common/network/connection_impl.cc:377] [C357] socket event: 3
[trace][connection] external/envoy/source/common/network/connection_impl.cc:445] [C357] write ready
[trace][connection] external/envoy/source/common/network/connection_impl.cc:415] [C357] read ready

jcurlier on Oct 5, 2018