lithops: Function call failed (500) - Retrying Request
Hi!
I am trying to run Lithops on an Azure-managed k8s cluster through Knative. My config file looks like the following:
lithops:
storage: "azure_blob"
storage_bucket: "storage"
serverless:
backend: "knative"
knative:
kubecfg_path: "/home/csegarra/.kube/config"
istio_endpoint: "http://52.224.128.238:80"
azure_blob:
storage_account: "lithops"
storage_account_key: <KEY>
As far as I can tell, knative works fine, and so does the storage (lithops installs the runtimes and job tmp files adequately). However, any execution will hang in the synchronisation point:
$ lithops test --debug
2021-03-08 08:53:29,386 [INFO] lithops.config -- Lithops v2.2.16
2021-03-08 08:53:29,386 [DEBUG] lithops.config -- Loading configuration
2021-03-08 08:53:29,389 [DEBUG] lithops.config -- Loading Serverless backend module: knative
2021-03-08 08:53:29,989 [DEBUG] lithops.config -- Loading Storage backend module: azure_blob
2021-03-08 08:53:30,267 [DEBUG] lithops.storage.backends.azure_blob.azure_blob -- Creating Azure Blob Storage client
2021-03-08 08:53:30,268 [INFO] lithops.storage.backends.azure_blob.azure_blob -- Azure Blob Storage client created
2021-03-08 08:53:30,316 [DEBUG] lithops.serverless.backends.knative.knative -- Loaded service host suffix: .default.52.224.128.238.xip.io
2021-03-08 08:53:30,316 [INFO] lithops.serverless.backends.knative.knative -- Knative client created - Istio Endpoint: http://52.224.128.238:80
2021-03-08 08:53:30,316 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Total available workers: 250
2021-03-08 08:53:30,319 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Serverless invoker created
2021-03-08 08:53:30,319 [INFO] lithops.executors -- Serverless Executor created with ID: 123d6b-0
2021-03-08 08:53:30,319 [INFO] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Selected Runtime: csegarragonz/lithops-knative-v38:2216 - 256MB
2021-03-08 08:53:30,320 [DEBUG] lithops.storage.storage -- Runtime metadata found in local cache
2021-03-08 08:53:30,320 [DEBUG] lithops.job.job -- ExecutorID 123d6b-0 | JobID A000 - Serializing function and data
2021-03-08 08:53:30,322 [DEBUG] lithops.job.serialize -- Referenced modules: {'/home/csegarra/.local/lib/python3.8/site-packages/lithops/scripts/cli.py'}
2021-03-08 08:53:30,323 [DEBUG] lithops.job.serialize -- Modules to transmit: None
2021-03-08 08:53:30,323 [INFO] lithops.job.job -- ExecutorID 123d6b-0 | JobID A000 - Uploading function and data - Total: 775.0B
2021-03-08 08:53:30,768 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Invoker process 0 started
2021-03-08 08:53:30,768 [INFO] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Starting function invocation: hello() - Total: 1 activations
2021-03-08 08:53:30,769 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Free workers: 250 - Going to invoke 1 function activations
2021-03-08 08:53:30,771 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Invoker process 1 started
2021-03-08 08:53:30,771 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:30,872 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Starting job monitoring
2021-03-08 08:53:30,873 [INFO] lithops.executors -- ExecutorID 123d6b-0 - Getting results
2021-03-08 08:53:30,965 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:30,979 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:31,141 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:34,147 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:34,309 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:39,317 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:39,480 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:43,485 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:43,648 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:48,655 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:48,816 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:49,849 [INFO] lithops.executors -- ExecutorID 123d6b-0 - Cancelled - Total Activations not done: 1
2021-03-08 08:53:49,849 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Stopping invoker
2021-03-08 08:53:49,849 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Invoker process 1 finished
Aborted!
and it would keep trying forever. Any ideas about what may be causing this issue?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (10 by maintainers)
It does! Thank you vey much for your help these days!
This should be fixed on master branch
I’ve finally been able to reproduce the issue, will submit a fix soon
Great! everything is looking good. I will investigate the possible causes of the 500 error and come back to you. thanks for the feedback.