google-auth-library-python: Max retries exceeded caused by SSLError(SSLZeroReturnError

Hi, Im getting this error since mid June when before it was working well.

Max retries exceeded with url: /entities/x?fields=y (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))

I’m getting this error for all the external sources I try to request from this slack bot. Being storage, bigquery, cloud runs and cloud functions. I want to emphasise that this had only started happening in mid June, and before it was working well.

When I run the same repo locally, it works perfectly. This is why I think it could be something related to auth? Sorry if not, I’m pretty new to this.

Any help would be much appreciated!

Environment details

Running on Cloud Run. Fast API and Slack SDK packages. Python Version: 3.11.4

Requirements.txt:

anyio==3.7.1
cachetools==5.3.1
certifi==2023.5.7
charset-normalizer==3.2.0
dealroom-firestore-connector
dealroom-urlextract
fastapi==0.78.0
filelock==3.12.2
google-api-core==2.11.1
google-api-core[grpc]==2.11.1
google-auth==2.22.0
google-cloud-bigquery==3.11.3
google-cloud-core==2.3.3
google-cloud-firestore==2.11.1
google-cloud-pubsub==2.18.0
google-cloud-storage==2.10.0
google-cloud==0.34.0
google-crc32c==1.5.0
google-resumable-media==2.5.0
googleapis-common-protos==1.59.1
googleapis-common-protos[grpc]==1.59.1
grpc-google-iam-v1==0.12.6
grpcio-status==1.56.0
grpcio==1.56.0
gunicorn==20.1.0
idna==3.4
numpy==1.25.1
packaging==23.1
pandas==2.0.3
proto-plus==1.22.3
protobuf==4.23.4
pyasn1-modules==0.3.0
pyasn1==0.5.0
pydantic==1.10.11
python-dateutil==2.8.2
pytz==2023.3
requests-file==1.5.1
requests==2.31.0
rsa==4.9
setuptools==68.0.0
six==1.16.0
slack-bolt-router==0.1.1
slack-bolt==1.18.0
slack-sdk==3.21.3
sniffio==1.3.0
starlette==0.19.1
tldextract==3.4.4
typing-extensions==4.7.1
tzdata==2023.3
urllib3==1.26.16

Full error report:

  File "/usr/local/lib/python3.9/site-packages/slack_bolt/listener/thread_runner.py", line 120, in run_ack_function_asynchronously
    listener.run_ack_function(request=request, response=response)
  File "/usr/local/lib/python3.9/site-packages/slack_bolt/listener/custom_listener.py", line 50, in run_ack_function
    return self.ack_function(
  File "/app/app/main.py", line 108, in verification_submission
    _verification_submission(ack, payload, body, say)
  File "/app/app/commands/modal_actions.py", line 169, in _verification_submission
    messages, not_found = process_verification_payload(USER, profiles, action)
  File "/app/app/utils.py", line 152, in process_verification_payload
    entity_uuid = get_entity(clean_profile_url)["uuid"]
  File "/app/app/services/db_connector.py", line 19, in get_entity
    res = _session.get(f"entities/{identifier}", params={"fields": "uuid"})
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/app/app/services/cloud_run_auth.py", line 299, in request
    return super(AuthorizedBaseUrlSession, self).request(
  File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 549, in request
    response = super(AuthorizedSession, self).request(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='db-connector-get-prod-x3nzdls4ha-ez.a.run.app', port=443): Max retries exceeded with url: /entities/verschuren_centre?fields=uuid (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))```


**Other errors related to this, that I also started getting at the same time:**

1_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

2_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 330, in _invoke_callbacks
    callback(self)
  File "/app/app/publish.py", line 18, in callback
    print(publish_future.result(timeout=60))
  File "/usr/local/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/futures.py", line 66, in result
    return super().result(timeout=timeout)
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 274, in _commit
    response = self._client._gapic_publish(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 267, in _gapic_publish
    return super().publish(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/pubsub_v1/services/publisher/client.py", line 831, in publish
    response = rpc(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 207, in retry_target
    raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 600.0s exceeded while calling target function, last exception: 504 Deadline Exceeded 

3_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /storage/v1/b/x/o/x.csv?fields=name&prettyPrint=false (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)'))) 

4_
 Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 482, in api_request
    response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request
    return self._do_request(
  File "/usr/local/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request
    return self.http.request(
  File "/usr/local/lib/python3.9/site-packages/google/auth/transport/requests.py", line 549, in request
    response = super(AuthorizedSession, self).request(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries exceeded with url: /bigquery/v2/projects/x/jobs?prettyPrint=false (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))

Thanks for your time!