azure-sdk-for-python: docker container with python azure-storage-blob sdk cannot connect to azure storage.
- azure-blob-storage:
- 12.12.0:
- Windows 11 (running ubuntu WSL2):
- python 3.8:
Describe the bug In a project where I am containerizing some of our organization’s python code, I am running into an issue whenever the python scripts attempt to connect to azure storage, I am faced with this error.
. . .
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 376, in send
raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f1354bd0eb0>: Failed to establish a new connection: [Errno -2] Name or service not known
An interesting fact, is we also have a .net console application running in the same container which uses the .net azure storage SDK and that application is able to connect to Azure storage without issue using the same connection string. The issue only occurs in the python code and only when running in the container. Running the code locally outside of the docker container does not produce this issue.
I was able to reproduce this issue in a separate docker container using only the azure storage library details below.
To Reproduce Steps to reproduce the behavior:
-
Create a directory structure to hold a dockerfile and python script code
-
Create a python script (py-console-az-storage/apps/main.py) with the following code
from azure.storage.blob import BlobServiceClient
import json
import os
def get_blob(container: str, blob:str):
global blobServiceClient
containerClient = blobServiceClient.get_container_client(container)
blobstr = containerClient.download_blob(blob).readall()
return json.loads(blobstr)
blobServiceClient: BlobServiceClient = BlobServiceClient.from_connection_string(os.getenv("AZURE_CONNECTION_STRING"))
container: str = os.getenv("container")
blob: str = os.getenv("blob")
print("Downloading BLOB method")
print(get_blob(container, blob))
- Create a Dockerfile (py-console-az-storage/Dockerfile) to containerize the script
FROM ubuntu:20.04
RUN mkdir -p /root/code/
WORKDIR /root/
RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y python3.8 python3.8-dev python3.8-venv python3.8-distutils
COPY . /root/code/
RUN echo "Setting Up Python environment.."
RUN python3.8 -m venv /root/env
ENV PATH=/root/env/bin:$PATH
RUN pip install pip==21.3.1
RUN pip install --no-cache-dir --upgrade -r /root/code/requirements.txt
- Create a requirements.txt file (py-console-az-storage/requirements.txt)
azure-storage-blob==12.12.0
-
Build the Dockerfile (navigate to the py-console-az-storage root folder) by running
docker image build --file ./Dockerfile --tag py-storage-console:1
-
Create an environment file with the following settings (py-console-az-storage/test.env)
AZURE_CONNECTION_STRING='[ Insert your storage account connection string here]'
container='[Name of a container that exists]'
blob='[path to json file within that contianer]'
-
Run the Docker container using this environment file
docker run -it --env-file ./test.env --entrypoint "python /root/code/app/main.py" py-storage-console:1
-
You will see thethe following output
PS C:\code\aks-volume-mount-example\apps\py-console-az-storage> docker run -it --env-file ./test.env py-storage-console:1
root@5891a613e549:~# python /root/code/app/main.py
Downloading BLOB method 1:
Traceback (most recent call last):
File "/root/code/app/main.py", line 33, in <module>
print(get_blob(container, blob))
File "/root/code/app/main.py", line 9, in get_blob
blobstr = containerClient.download_blob(blob).readall()
File "/root/env/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_container_client.py", line 1151, in download_blob
return blob_client.download_blob(offset=offset, length=length, **kwargs)
File "/root/env/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py", line 868, in download_blob
return StorageStreamDownloader(**options)
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_download.py", line 355, in __init__
self._response = self._initial_request()
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_download.py", line 392, in _initial_request
location_mode, response = self._clients.blob.download(
File "/root/env/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
return func(*args, **kwargs)
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/_blob_operations.py", line 1669, in download
pipeline_response = self._client._pipeline.run( # pylint: disable=protected-access
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 205, in run
return first_node.send(pipeline_request) # type: ignore
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
[Previous line repeated 2 more times]
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py", line 160, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 543, in send
raise err
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 517, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
[Previous line repeated 1 more time]
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 313, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
response = self.next.send(request)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 100, in send
self._sender.send(request.http_request, **request.context.options),
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 333, in send
return self._transport.send(request, **kwargs)
File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 333, in send
return self._transport.send(request, **kwargs)
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 376, in send
raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f1354bd0eb0>: Failed to establish a new connection: [Errno -2] Name or service not known
Expected behavior This error should not be happening as the connection string, storage container name and blob path are valid. The code should download the file and output it to the terminal like it does when running this script locally outside of the docker image.
Additional context I’ve seen other users having similar issues on this forum but I have not found a working solution.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 27 (17 by maintainers)
I figured out what was going on… In my env file, I have the values formatted like this
If I remove the single quotes from each environment variable the connection works. I did not realize having these values encapsulated in quote marks passes the literal value of the quote character as the variable’s value.
Removing the quotes works when running the container with an environment file.
@xiangyan99 @jalauzon-msft Thank you for your assistance in troubleshooting this issue with me
@jalauzon-msft does this work? https://github.com/moby/moby/issues/30757#issuecomment-283304977
Hi @Srfigie, thank you for opening an issue! I’ll tag some folks who should be able to help and we’ll get back to you as soon as we can.