azure-sdk-for-java: [BUG] Listing blobs with a special character in the blob name fails with 403 AuthenticationFailed

Describe the bug Listing blobs with a special character in the blob name fails. If we encode the prefix then there is no exception but the listing returns 0 objects. Authentication is via SharedKeyCredential

Exception or Stack Trace

Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:3850e592-601e-00a7-45f2-bbbb36000000
Time:2020-11-16T08:25:51.2139806Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'hMA5oVOIWCMUj/qjM0es8z9ALvkUM1MFC/Dg/htyHvo=' is not the same as any computed signature. Server used following string to sign: 'GET
Mon, 16 Nov 2020 08:25:51 GMT
x-ms-client-request-id:cc000ea8-05b7-4380-aaea-de2ac67c41c4
x-ms-version:2020-02-10
/kompriseqalrs/komprise-somansh
comp:list
delimiter:/
prefix:tmp/ks/1059/Special Files/files with period/weird/JPC/Grants, reports etc/etcfile.txt
restype:container'.</AuthenticationErrorDetail></Error>"

To Reproduce List Blobs with a prefix that has a special non ASCII character like tmp/ks/1059/Special Files/files with period/weird/JPC/Grants, reports etc/etcfile.txt

Code Snippet

ListBlobsOptions options = new ListBlobsOptions().setPrefix(prefix);
PagedIterable<BlobItem> pagedIterable =  recursive ? container.listBlobs(options, TIMEOUT)
                                                   : container.listBlobsByHierarchy(/*delimiter*/ "/", options, TIMEOUT);
pagedIterable.forEach(action);  // action is a consumer which just adds blobs to a list

Expected behavior List one blob which is the current blob itself.

This works in v4.0.0

for (ListBlobItem blobItem : container.listBlobs(keyPrefix, recursive, EnumSet.noneOf(BlobListingDetails.class), /*options*/ null, operationContext))

Screenshots None

Setup:

  • OS: Ubuntu 18.04
  • IDE : IntelliJ 19.1.4
  • Version of the Library used - azure-storage-blob v12.7.0 ( issue seen even in azure-storage-blob v12.9.0-beta.1 and 12.9.0 but would like a workaround for 12.7.0 if possible )

Additional context On a side issue, creating a blob client for blob names with % fails. I presume this is due to the fact that the Azure SDK does a Url decode internally like Utility.urlEncode(Utility.urlDecode(blobName))) which gives the below stack trace for blob name Special Files/SingleSpecialCharactersInFileName/.TestFile_%636940838567587622_1%.md

Caused by: java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: ".m"
	at java.net.URLDecoder.decode(URLDecoder.java:194) ~[na:1.8.0_252]
	at com.azure.storage.common.Utility.decode(Utility.java:99) ~[kcfscloud-3.2.0.jar:na]
	at com.azure.storage.common.Utility.urlDecode(Utility.java:90) ~[kcfscloud-3.2.0.jar:na]
	at com.azure.storage.blob.BlobContainerAsyncClient.getBlobAsyncClient(BlobContainerAsyncClient.java:170) ~[kcfscloud-3.2.0.jar:na]
	at com.azure.storage.blob.BlobContainerAsyncClient.getBlobAsyncClient(BlobContainerAsyncClient.java:151) ~[kcfscloud-3.2.0.jar:na]
	at com.azure.storage.blob.BlobContainerClient.getBlobClient(BlobContainerClient.java:80) ~[kcfscloud-3.2.0.jar:na]

A workaround for this was to encode the % before passing it to the SDK. So we re-used the SDK’s built in Url encode Utility.urlEncode()

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

As for your additional context part, yes the SDK expects a user to encode their URL if it has any special characters.

Just to be clear for anyone else experiencing this problem, from my tests I have concluded you should URL encode the blob name when creating a blob, but you must not URL encode the prefix when using listBlobsByHiererchy or it will return nothing as @somanshreddy pointed out initially.

Also, when URL encoding the blob name on create, you cannot use java.net.URLEncoder because it encodes a space to “+”. Azure requires that a space be URL encoded to “%20”. This is a not specific to Java because the exact same thing happens in the .Net Azure SDK.

Yes, this should be released soon. I will update this post as soon as it goes out!

@somanshreddy

The updated SDK should be out now! Should be azure-storage-blob 12.10.0

@somanshreddy It’s definitely a bug internal to the SDK so you will have to wait until our next release (likely next month)

@somanshreddy Thanks for following up. I was able to repro the issue and I am looking into the best way to fix it.

@somanshreddy Thanks for providing the repro code.

Sorry, the code I shared was testing the Java SDK - we use a framework called Groovy to run tests, so it looks a little different.

I will work on reproducing with the code snippet you provided.