lithops: Function call failed (500) - Retrying Request

Hi!

I am trying to run Lithops on an Azure-managed k8s cluster through Knative. My config file looks like the following:

lithops:
    storage: "azure_blob"
    storage_bucket: "storage"

serverless:
    backend: "knative"

knative:
    kubecfg_path: "/home/csegarra/.kube/config"
    istio_endpoint: "http://52.224.128.238:80"

azure_blob:
    storage_account: "lithops"
    storage_account_key:  <KEY>

As far as I can tell, knative works fine, and so does the storage (lithops installs the runtimes and job tmp files adequately). However, any execution will hang in the synchronisation point:

$ lithops test --debug
2021-03-08 08:53:29,386 [INFO] lithops.config -- Lithops v2.2.16
2021-03-08 08:53:29,386 [DEBUG] lithops.config -- Loading configuration
2021-03-08 08:53:29,389 [DEBUG] lithops.config -- Loading Serverless backend module: knative
2021-03-08 08:53:29,989 [DEBUG] lithops.config -- Loading Storage backend module: azure_blob
2021-03-08 08:53:30,267 [DEBUG] lithops.storage.backends.azure_blob.azure_blob -- Creating Azure Blob Storage client
2021-03-08 08:53:30,268 [INFO] lithops.storage.backends.azure_blob.azure_blob -- Azure Blob Storage client created
2021-03-08 08:53:30,316 [DEBUG] lithops.serverless.backends.knative.knative -- Loaded service host suffix: .default.52.224.128.238.xip.io
2021-03-08 08:53:30,316 [INFO] lithops.serverless.backends.knative.knative -- Knative client created - Istio Endpoint: http://52.224.128.238:80
2021-03-08 08:53:30,316 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Total available workers: 250
2021-03-08 08:53:30,319 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Serverless invoker created
2021-03-08 08:53:30,319 [INFO] lithops.executors -- Serverless Executor created with ID: 123d6b-0
2021-03-08 08:53:30,319 [INFO] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Selected Runtime: csegarragonz/lithops-knative-v38:2216 - 256MB 
2021-03-08 08:53:30,320 [DEBUG] lithops.storage.storage -- Runtime metadata found in local cache
2021-03-08 08:53:30,320 [DEBUG] lithops.job.job -- ExecutorID 123d6b-0 | JobID A000 - Serializing function and data
2021-03-08 08:53:30,322 [DEBUG] lithops.job.serialize -- Referenced modules: {'/home/csegarra/.local/lib/python3.8/site-packages/lithops/scripts/cli.py'}
2021-03-08 08:53:30,323 [DEBUG] lithops.job.serialize -- Modules to transmit: None
2021-03-08 08:53:30,323 [INFO] lithops.job.job -- ExecutorID 123d6b-0 | JobID A000 - Uploading function and data - Total: 775.0B
2021-03-08 08:53:30,768 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Invoker process 0 started
2021-03-08 08:53:30,768 [INFO] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Starting function invocation: hello() - Total: 1 activations
2021-03-08 08:53:30,769 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Free workers: 250 - Going to invoke 1 function activations
2021-03-08 08:53:30,771 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Invoker process 1 started
2021-03-08 08:53:30,771 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:30,872 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 | JobID A000 - Starting job monitoring
2021-03-08 08:53:30,873 [INFO] lithops.executors -- ExecutorID 123d6b-0 - Getting results
2021-03-08 08:53:30,965 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:30,979 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:31,141 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:34,147 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:34,309 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:39,317 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:39,480 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:43,485 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:43,648 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:48,655 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Invoking function call 00000
2021-03-08 08:53:48,816 [DEBUG] lithops.serverless.backends.knative.knative -- ExecutorID 123d6b-0 | JobID A000 - Function call 00000 failed (500). Retrying request
2021-03-08 08:53:49,849 [INFO] lithops.executors -- ExecutorID 123d6b-0 - Cancelled - Total Activations not done: 1
2021-03-08 08:53:49,849 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Stopping invoker
2021-03-08 08:53:49,849 [DEBUG] lithops.invokers -- ExecutorID 123d6b-0 - Invoker process 1 finished

Aborted!

and it would keep trying forever. Any ideas about what may be causing this issue?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (10 by maintainers)

Most upvoted comments

It does! Thank you vey much for your help these days!

This should be fixed on master branch

I’ve finally been able to reproduce the issue, will submit a fix soon

Great! everything is looking good. I will investigate the possible causes of the 500 error and come back to you. thanks for the feedback.