azure-sdk-for-python: Azure DL Storage Gen2 runs into timeout on get_paths

Package Name: from azure.storage.filedatalake import DataLakeServiceClient
Package Version: azure-storage-file-datalake==12.0.0b6
Operating System: Unix/Debian/Ubuntu
Python Version: Python 3.7.3

Describe the bug [1]

service_client = DataLakeServiceClient(account_url=accountURL, credential=storageKey)
fsClient = service_client.get_file_system_client(file_system=filesystem)
        paths = fsClient.get_paths(path=directory)

When running this code on ADLS Gen2 I run into a timeout:

azure.storage.filedatalake._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'Operation could not be completed within the specified time.'

Same thing happens sometimes when trying to view the content of the DL in Storage explorer. <?xml version="1.0" encoding="utf-8"?><Error><Code>OperationTimedOut</Code><Message>Operation could not be completed within the specified time. RequestId:c529aebd-e01e-0063-0ab7-f66173000000...

The datalake folder containing avro files from EventHub has about 60 pages and 5,986 Elements

To Reproduce Steps to reproduce the behavior:

run the snippet from above [1] to a datalake gen2 with about 5,986 elements with an element size from 508B to 132KB

Expected behavior The generator with paths from the DL folder should be created in variable paths

Screenshots Class Method in my actual program Traceback when trying to debug on command line.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16

Most upvoted comments

Request ID is RequestId:31f106e9-101f-00fe-6a78-f71b33000000

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/benjamin/.local/lib/python3.7/site-packages/azure/core/paging.py", line 122, in __next__
    return next(self._page_iterator)
  File "/home/benjamin/.local/lib/python3.7/site-packages/azure/core/paging.py", line 74, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/home/benjamin/.local/lib/python3.7/site-packages/azure/storage/filedatalake/_models.py", line 298, in _get_next_cb
    cls=return_headers_and_deserialized_path_list)
  File "/home/benjamin/.local/lib/python3.7/site-packages/azure/storage/filedatalake/_generated/operations/_file_system_operations.py", line 442, in list_paths
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.filedatalake._generated.models._models_py3.StorageErrorException: (OperationTimedOut) Operation could not be completed within the specified time.
RequestId:31f106e9-101f-00fe-6a78-f71b33000000
Time:2020-03-11T07:38:48.0289622Z

damnedOperator on Mar 11, 2020