jina: Endless Waiting executor0 but the service works normally.
the code to run from #5959
DEBUG executor0/rep-0@39476 Setting signal handlers [07/14/23 08:58:52]
DEBUG executor0/rep-0@39476 Signal handlers already set
DEBUG executor0-replica-set@39272 Waiting for ReplicaSet to start successfully [07/14/23 08:58:52]
⠋ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 -:--:--DEBUG gateway/rep-0@39477 Setting signal handlers [07/14/23 08:58:52]
DEBUG gateway-replica-set@39272 Waiting for ReplicaSet to start successfully [07/14/23 08:58:52]
⠋ Waiting ... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 -:--:--DEBUG gateway/rep-0@39477 Signal handlers already set
DEBUG gateway/rep-0@39477 adding connection for deployment executor0/heads/0 to grpc://0.0.0.0:62917 [07/14/23 08:58:52]
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to grpc://0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to grpc://0.0.0.0:62917
DEBUG gateway/rep-0@39477 connection for deployment executor0/heads/0 to grpc://0.0.0.0:62917 added
DEBUG gateway/rep-0@39477 Setting up GRPC server
DEBUG gateway/rep-0@39477 Get all endpoints from TopologyGraph
DEBUG gateway/rep-0@39477 Running GatewayRuntime warmup
DEBUG gateway/rep-0@39477 Getting Endpoints data from executor0
DEBUG gateway/rep-0@39477 starting warmup task for deployment executor0
DEBUG gateway/rep-0@39477 gRPC call to executor0 for EndpointDiscovery errored, with error <AioRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:58:52.240257381+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection
refused {grpc_status:14, created_time:"2023-07-14T08:58:52.240253323+08:00"}]}"
> and for the 1th time.
DEBUG gateway/rep-0@39477 resetting connection for executor0 to 0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to 0.0.0.0:62917
⠹ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:01DEBUG gateway/rep-0@39477 gRPC call to executor0 for EndpointDiscovery errored, with error <AioRpcError of RPC that terminated with: [07/14/23 08:58:53]
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:58:53.241945446+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection
refused {grpc_status:14, created_time:"2023-07-14T08:58:53.241944256+08:00"}]}"
> and for the 2th time.
DEBUG gateway/rep-0@39477 resetting connection for executor0 to 0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to 0.0.0.0:62917
⠦ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:02DEBUG gateway/rep-0@39477 gRPC call to executor0 for EndpointDiscovery errored, with error <AioRpcError of RPC that terminated with: [07/14/23 08:58:55]
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:58:55.150302259+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection
refused {created_time:"2023-07-14T08:58:55.150299709+08:00", grpc_status:14}]}"
> and for the 3th time.
DEBUG gateway/rep-0@39477 resetting connection for executor0 to 0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to 0.0.0.0:62917
⠧ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:05DEBUG gateway/rep-0@39477 gRPC call to executor0 for EndpointDiscovery errored, with error <AioRpcError of RPC that terminated with: [07/14/23 08:58:57]
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:58:57.65715233+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused
{created_time:"2023-07-14T08:58:57.657149449+08:00", grpc_status:14}]}"
> and for the 4th time.
DEBUG gateway/rep-0@39477 gRPC call for executor0 failed, retries exhausted
DEBUG gateway/rep-0@39477 resetting connection for executor0 to 0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to 0.0.0.0:62917
WARNI… gateway/rep-0@39477 Getting endpoints failed: failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused. Waiting for another trial
⠏ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:06DEBUG gateway/rep-0@39477 Getting Endpoints data from executor0 [07/14/23 08:58:58]
⠙ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:09DEBUG gateway/rep-0@39477 gRPC call to executor0 for EndpointDiscovery errored, with error <AioRpcError of RPC that terminated with: [07/14/23 08:59:01]
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:59:01.975496864+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection
refused {grpc_status:14, created_time:"2023-07-14T08:59:01.975494113+08:00"}]}"
> and for the 1th time.
DEBUG gateway/rep-0@39477 resetting connection for executor0 to 0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to 0.0.0.0:62917
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:11<00:00, 1.67s/it]
⠦ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:17DEBUG gateway/rep-0@39477 gRPC call to executor0 for EndpointDiscovery errored, with error <AioRpcError of RPC that terminated with: [07/14/23 08:59:09]
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:59:09.637254313+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection
refused {grpc_status:14, created_time:"2023-07-14T08:59:09.637251379+08:00"}]}"
> and for the 2th time.
⠇ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:17DEBUG gateway/rep-0@39477 resetting connection for executor0 to 0.0.0.0:62917
DEBUG gateway/rep-0@39477 create_connection connection for executor0 to 0.0.0.0:62917
⠋ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:19DEBUG executor0/rep-0@39476 <__main__.GLMInference object at 0x7f129e307f40> is successfully loaded! [07/14/23 08:59:11]
DEBUG executor0/rep-0@39476 Setting up GRPC server
INFO executor0/rep-0@39476 start server bound to 0.0.0.0:62917
DEBUG executor0/rep-0@39476 server bound to 0.0.0.0:62917 started
DEBUG executor0/rep-0@39476 GRPC server setup successful
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC [07/14/23 08:59:11]
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:12]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:12.556621633+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:13]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:13.666544143+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:14]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:14.774546914+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:15]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:15.882533002+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:16]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:16.990531376+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC [07/14/23 08:59:17]
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:18]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:18.098532957+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:19]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:19.206557554+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:20]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:20.313535613+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
⠼ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:29DEBUG executor0/rep-0@39476 got an endpoint discovery request [07/14/23 08:59:21]
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:21]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:21.421539286+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
⠼ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:29DEBUG gateway/rep-0@39477 Got all endpoints from TopologyGraph {'/chat', '/chat_manas', '_jina_dry_run_'} [07/14/23 08:59:21]
INFO gateway/rep-0@39477 start server bound to 0.0.0.0:19002
DEBUG gateway/rep-0@39477 server bound to 0.0.0.0:19002 started
DEBUG gateway/rep-0@39477 GRPC server setup successful
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG gateway/rep-0@39272 ready and listening [07/14/23 08:59:21]
DEBUG gateway-replica-set@39272 ReplicaSet started successfully [07/14/23 08:59:21]
DEBUG gateway@39272 Deployment started successfully [07/14/23 08:59:21]
⠦ Waiting executor0 gateway... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/2 0:00:29DEBUG executor0/rep-0@39476 recv _status request
DEBUG gateway/rep-0@39477 completed warmup task in 29.353113174438477s.
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:22]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:22.529546379+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:23]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:23.636531418+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:25]
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: HTTP proxy returned response code 407"
debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2023-07-14T08:59:25.646676539+08:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: HTTP proxy returned response code 407
{created_time:"2023-07-14T08:59:25.646673591+08:00", grpc_status:14}]}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:26]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:26.754537615+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:27]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:27.863538749+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:28]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {grpc_status:4, created_time:"2023-07-14T08:59:28.971538289+08:00"}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC [07/14/23 08:59:29]
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:30]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:30.078544313+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:31]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:31.185541927+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
DEBUG executor0/rep-0@39272 Checking readiness to 0.0.0.0:62917 with protocol GRPC
DEBUG executor0/rep-0@39272 Exception: <AioRpcError of RPC that terminated with: [07/14/23 08:59:32]
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-07-14T08:59:32.29353679+08:00", grpc_status:4}"
>
DEBUG executor0/rep-0@39272 Server on 0.0.0.0:62917 with protocol GRPC is not yet ready
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 41 (21 by maintainers)
Thank you very much for your help for a long time.
I found something on stackoverflow
unset https_proxy;unset http_proxy, It solved this problem. @JoanFM