azure-sdk-for-python: docker container with python azure-storage-blob sdk cannot connect to azure storage.

  • azure-blob-storage:
  • 12.12.0:
  • Windows 11 (running ubuntu WSL2):
  • python 3.8:

Describe the bug In a project where I am containerizing some of our organization’s python code, I am running into an issue whenever the python scripts attempt to connect to azure storage, I am faced with this error.

. . .
File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 376, in send
    raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f1354bd0eb0>: Failed to establish a new connection: [Errno -2] Name or service not known

An interesting fact, is we also have a .net console application running in the same container which uses the .net azure storage SDK and that application is able to connect to Azure storage without issue using the same connection string. The issue only occurs in the python code and only when running in the container. Running the code locally outside of the docker container does not produce this issue.

I was able to reproduce this issue in a separate docker container using only the azure storage library details below.

To Reproduce Steps to reproduce the behavior:

  1. Create a directory structure to hold a dockerfile and python script code image

  2. Create a python script (py-console-az-storage/apps/main.py) with the following code

from azure.storage.blob import BlobServiceClient
import json
import os

def get_blob(container: str, blob:str):
    global blobServiceClient

    containerClient = blobServiceClient.get_container_client(container)
    blobstr = containerClient.download_blob(blob).readall()
    
    return json.loads(blobstr)

blobServiceClient: BlobServiceClient = BlobServiceClient.from_connection_string(os.getenv("AZURE_CONNECTION_STRING"))
container: str = os.getenv("container")
blob: str = os.getenv("blob")

print("Downloading BLOB method")
print(get_blob(container, blob))
  1. Create a Dockerfile (py-console-az-storage/Dockerfile) to containerize the script
FROM ubuntu:20.04

RUN mkdir -p /root/code/
WORKDIR /root/

RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y python3.8 python3.8-dev python3.8-venv python3.8-distutils 

COPY . /root/code/

RUN echo "Setting Up Python environment.."
RUN python3.8 -m venv /root/env
ENV PATH=/root/env/bin:$PATH
RUN pip install pip==21.3.1 

RUN pip install --no-cache-dir --upgrade -r /root/code/requirements.txt
  1. Create a requirements.txt file (py-console-az-storage/requirements.txt)
azure-storage-blob==12.12.0
  1. Build the Dockerfile (navigate to the py-console-az-storage root folder) by running docker image build --file ./Dockerfile --tag py-storage-console:1

  2. Create an environment file with the following settings (py-console-az-storage/test.env)

AZURE_CONNECTION_STRING='[ Insert your storage account connection string here]'
container='[Name of a container that exists]'
blob='[path to json file within that contianer]'
  1. Run the Docker container using this environment file docker run -it --env-file ./test.env --entrypoint "python /root/code/app/main.py" py-storage-console:1

  2. You will see thethe following output

PS C:\code\aks-volume-mount-example\apps\py-console-az-storage> docker run -it --env-file ./test.env py-storage-console:1
root@5891a613e549:~# python /root/code/app/main.py
Downloading BLOB method 1:
Traceback (most recent call last):
  File "/root/code/app/main.py", line 33, in <module>
    print(get_blob(container, blob))
  File "/root/code/app/main.py", line 9, in get_blob
    blobstr = containerClient.download_blob(blob).readall()
  File "/root/env/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_container_client.py", line 1151, in download_blob
    return blob_client.download_blob(offset=offset, length=length, **kwargs)
  File "/root/env/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_blob_client.py", line 868, in download_blob
    return StorageStreamDownloader(**options)
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_download.py", line 355, in __init__
    self._response = self._initial_request()
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_download.py", line 392, in _initial_request
    location_mode, response = self._clients.blob.download(
  File "/root/env/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_generated/operations/_blob_operations.py", line 1669, in download
    pipeline_response = self._client._pipeline.run(  # pylint: disable=protected-access
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 205, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  [Previous line repeated 2 more times]
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py", line 160, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 543, in send
    raise err
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 517, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/policies.py", line 313, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 100, in send
    self._sender.send(request.http_request, **request.context.options),
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 333, in send
    return self._transport.send(request, **kwargs)
  File "/root/env/lib/python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 333, in send
    return self._transport.send(request, **kwargs)
  File "/root/env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 376, in send
    raise error
azure.core.exceptions.ServiceRequestError: <urllib3.connection.HTTPSConnection object at 0x7f1354bd0eb0>: Failed to establish a new connection: [Errno -2] Name or service not known

Expected behavior This error should not be happening as the connection string, storage container name and blob path are valid. The code should download the file and output it to the terminal like it does when running this script locally outside of the docker image.

Additional context I’ve seen other users having similar issues on this forum but I have not found a working solution.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 27 (17 by maintainers)

Most upvoted comments

I figured out what was going on… In my env file, I have the values formatted like this

AZURE_CONNECTION_STRING='[ Insert your storage account connection string here]'
container='[Name of a container that exists]'
blob='[path to json file within that container]'

If I remove the single quotes from each environment variable the connection works. I did not realize having these values encapsulated in quote marks passes the literal value of the quote character as the variable’s value.

Removing the quotes works when running the container with an environment file.

AZURE_CONNECTION_STRING=[ Insert your storage account connection string here]
container=[Name of a container that exists]
blob=[path to json file within that container]

@xiangyan99 @jalauzon-msft Thank you for your assistance in troubleshooting this issue with me

Hi @Srfigie, thank you for opening an issue! I’ll tag some folks who should be able to help and we’ll get back to you as soon as we can.