azure-storage-azcopy: azcopy sync fails with "AuthenticationErrorDetail: se is mandatory", however the se is definitely included

After more testing I eventually got SAS tokens from AZ CLI to work.

EXPIRE=$(date -u -d "1 day" '+%Y-%m-%dT%H:%M:%SZ')
START=$(date -u -d "-1 day" '+%Y-%m-%dT%H:%M:%SZ')
ACCOUNT="some_account"
KEY="some_key"
CONTAINER="some_container"
BLOB="test.jpg"
SAS=$(az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --start "$START" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt | sed 's/%3A/:/g;s/\"//g')

azcopy copy "https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB}?${SAS}" "$HOME/test.jpg"

The Azure Node.js SDK also generates the SAS token with time format yyyy-MM-ddTHH:mm:ss.fffffffZ. And azcopy copy doesn’t work. However, azcopy make works well. They are using different method to parse the time?

Part of the bug is also the documentation of the az tool. It specifies:

    --expiry              : Specifies the UTC datetime (Y-m-d'T'H:M'Z') at which the SAS becomes
                            invalid. Do not use if a stored access policy is referenced with --id
                            that specifies this value.

But that format is not that strict. If I use Y-m-d’T’H:M:S’Z’ instead (%Y-%m-%dT%H:%M:%SZ in strftime) it works correctly.

As fate would have it, I just figured it out with help from another github post on the issue (sorry I can’t find it now to credit the author). It was as simple as including a literal in the date format for ‘00Z’:

NOW=`date +"%Y-%m-%dT%H:%M:00Z"`
EXPIRY=`date -d "$NOW + 10 years" +"%Y-%m-%dT%H:%M:00Z"`
SASa=`az storage container generate-sas --name $srcCon --start $NOW --expiry $EXPIRY --permissions rl --account-name $srcAcc --account-key $srcKey
--output tsv`

I do not know that this is exactly what will work for others but after many hours of hair-pulling it’s worked for me.

@zezha-msft thanks for the link just now. Doesn’t seem to load, but thanks!

I spent way more time digging into this than I expected last night. I was able to produce a working local version of azcopy by making changes purely to Blob Storage Go SDK. It looks like azcopy has forks of some of the files I was changing, but me making my edits there had seemingly no effect. Unsure if it’s dead code, or hidden behind a switch.

I’m going to try to prepare a PR for the Go SDK this week, to help speed things along. It’ll be my first change to a Go codebase! 💪

I’m having a similar issue. The REST API allows SAS tokens to be generated that work in other tools (e.g. Storage Explorer), but result in “Authentication Failed” when using AzCopy.

To replicate, generate some SAS tokens via the “Try It” option here: https://docs.microsoft.com/en-us/rest/api/storagerp/storageaccounts/listaccountsas

This caught me out as I’m generating a SAS token from an ARM template using the listAccountSas function, and passing it into a container as an environment variable which is later used by AzCopy.

Here’s the payload I’m using to generate a SAS token:

{
        "signedServices": "b",
        "signedPermission": "w",
        "signedProtocol": "https",
        "signedStart": "2019-01-01T00:00Z",
        "signedExpiry": "2029-01-01T00:00Z",
        "signedResourceTypes": "o",
        "keyToSign": "key1"
}

And here’s an example of how I am trying to use it (real resources, but now nuked):

azcopy cp "./test.xml" "https://k5h7iywh3tnvy.blob.core.windows.net/reports/test.xml?sv=2015-04-05&ss=b&srt=o&sp=w&st=2019-01-01T00%3A00%3A00.0000000Z&se=2029-01-01T00%3A00%3A00.0000000Z&spr=https&sig=%2BadXJbjjuULC9g8ryNzoQKccX0c6h3gVgAr7ESd8Bu0%3D"

Things I’ve noticed:

• The SAS token DOES work when you manually make the PUT using a tool like Postman, so the SAS token is valid.
• In Postman, you can decode all the URL entries (%3A is a colon in the time, for example) without it affecting the signature. The SAS token still works.
• You CANNOT drop the case of the T or Z in the date format. In the AzCopy failure output, timestamps are reported with lowercase T and Z, but it’s not clear whether that’s just a facet of the console output rather than an actual representation of the PUT operation.
• The st and se values in the SAS token have the values: “2019-01-01T11:11:11.0000000Z”. The sub-second precision is important: if you try to edit/remove the second component (even though they are all zeroes), then you invalidate the signature causing “AuthenticationFailed - Signature fields not well formed”.

So, while the SAS token APIs (accessible via REST or AZ CLI) are quite happy to generate tokens with different date formats, I suspect that AzCopy isn’t handling them verbatim and is breaking the signature as a result.

So, it turns out the tokens generated are not complete! The SAS token timestamps are wrong and this endpoint expects the following format:

?sv=2017-07-29&ss=b&srt=sco&sp=rwdlacup&se=2039-01-01T08:00:00Z&st=2018-01-01T08:00:00Z&spr=https&sig=<redacted>

Notice: &se=2028-01-01 &se=2039-01-01T08:00:00Z

Unfortunately the current version of this tool does not know how to parse the URL correctly and figure out what’s wrong.

Ok, sorted it out: Oh, I was running it from .cmd file on windows, thus lost all single % characters in SAS… Need to double them inside .cmd file, obviously. Totally my bad… 😦

---

az storage account generate-sas produce url encoded SAS “st=2021-01-01T00 %3A 00 %3A 00Z&se=2022-01-01T00 %3A 00 %3A 00Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=bla-bla-bla %3D”

and azcopy expects decoded one: "st=2021-01-01T00 : 00 : 00Z&se=2022-01-01T00 : 00 : 00Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=bla-bla-bla = "

• I put extra spaces around encoded chars above

azcopy shows funny errors to cheer you up and make you think! Ingenious!

How do I tell azcopy NOT to drop se value from SAS? The SAS has it, but clearly missing in PUT statement below. And very same question about st value

Time:2021-01-22T18:14:09.2117938Z, Details:
   AuthenticationErrorDetail: se is mandatory. Cannot be empty
   Code: AuthenticationFailed
   PUT https://-REDACTED-.blob.core.windows.net/-REDACTED-?blockid=nmm1ngvjogqtmzeyzs03yzq4ltu4yzitzdvjzmm4yjqymtbl&comp=block&sig=-REDACTED-&sp=rwdlacup&srt=sco&ss=b&sv=2018-03-28&timeout=901

I needed to use azcopy in Azure DevOps Pipelines but stuggled a lot to get it to work.

After trying out most suggestions, the only thing that worked for me was what @lmeyerov suggested, however, I don’t need to use the --account-key parameter.

For others that find this issue, I did it with:

  - task: Bash@3
    displayName: Install azcopy
    inputs:
      targetType: 'inline'
      script: |
        mkdir $(Agent.ToolsDirectory)/azcopy && cd "$_"
        wget -O azcopy_v10.tar.gz https://aka.ms/downloadazcopy-v10-linux
        tar -xf azcopy_v10.tar.gz --strip-components=1
  - task: AzureCLI@2
    displayName: Download using azcopy
    inputs:
      azureSubscription: my-vmssagents-service-connection
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        export STORE_NAME="data"
        export CONTAINER_NAME="data"

        NOW=`date +"%Y-%m-%dT%H:%M:00Z"` \
        EXPIRY=`date -d "$NOW + 1 day" +"%Y-%m-%dT%H:%M:00Z"` \
        && export SAS_TOKEN=$( az storage container generate-sas \
            --account-name $STORE_NAME \
            --name $CONTAINER_NAME \
            --start $NOW \
            --expiry $EXPIRY \
            --permissions acdlrw \
            --output tsv )

        $(Agent.ToolsDirectory)/azcopy/azcopy copy \
            "https://${STORE_NAME}.blob.core.windows.net/${CONTAINER_NAME}/${{ parameters.folder }}/?${SAS_TOKEN}" \
            "." --recursive --include-pattern "*c_*b.nc;left.nc;right.nc" # <-- my specific pattern

I am having the same problem with azure-storage-deploy and can’t figure out what the correct format is.

I have tried

&se=2028-01-01
&se=2039-01-01T08:00:00Z

Still, there is no hope!

The actual PUT URL is missing the se and st query parameters.

PUT https://XXXX.blob.core.windows.net/$web/webpack-runtime-887b0243a9c4b83c6243.js?sig=-REDACTED-&sp=rwdlacu&spr=https&srt=o&ss=b&sv=2017-07-29&timeout=901

It works locally with AzCopy 10.5.0 just fine.

Ist there any workaround in the meantime?

@lflemingweb I don’t think there’s any way to get the REST API to produce a working sas. The problem is that even if you pass it a datetime that works with azcopy, the API will return a sas that includes decimals in the seconds field, which fails.

Has anyone managed to get SAS tokens generated from AZ CLI to work with ‘azcopy copy’

I’ve tried generating tokens with all of the mentioned datetime formats mentioned here as well as the format from the AZ CLI docs and azcopy spits them all back.

Examples tried:

EXPIRE=`date -u -d "1 day" '+%FT%TZ'`

EXPIRE=`date -v+30M '+%Y-%m-%dT%H:%MZ'`

NOW=`date +"%Y-%m-%dT%H:%M:00Z"`
EXPIRE=`date -d "$NOW + 10 years" +"%Y-%m-%dT%H:%M:00Z"`

Using this AZ command to generate the SAS:

az storage account generate-sas --account-name "$ACCOUNT" --account-key "$KEY" --expiry "$EXPIRE" --https-only --permissions acdlpruw --resource-types sco --services bfqt

For reference I’ve compared a valid SAS token generated via the Azure Portal and a token generated from AZ CLI with %3A changed to : and they are more or less identical in formatting so this is very confusing as to what is going on with azcopy.

The only difference I can see is firstly the parameters of the SAS token are in a different order (Shouldn’t matter though) e.g portal generated token starts with sv= where as Az CLI token starts with st= and secondly the sig= is always a few characters longer when using portal generated tokens.

NOTE: These are not valid tokens I’ve changed the values in them so don’t worry about redacting Portal Generated Token: sv=2018-03-28&ss=bfqt&srt=sco&sp=rwdlacup&se=2020-09-06T07:00:52Z&st=2019-09-05T23:00:52Z&spr=https&sig=Xl4i%2Bsy%2Bg4ab7LstIguIlfXdOnhS6Slkdxe1CH0ptYE%3D

Az CLI Generated Token (With re-arranging): sv=2018-03-28&ss=bqtf&srt=sco&sp=rwdlacup&se=2021-01-09T21:35:42Z&st=2019-01-07T21:35:42Z&spr=https&sig=GkJC3PiP6NHa/QsNoIYMI/miHbLVdsIOCPinRJePKKg%3D

Az CLI Generated Token (Without re-arranging): st=2019-01-07T21:35:42Z&se=2021-01-09T21:35:42Z&sp=rwdlacup&spr=https&sv=2018-03-28&ss=bqtf&srt=sco&sig=GkJC3PiP6NHa/QsNoIYMI/miHbLVdsIOCPinRJePKKg%3D

I opened a ticket with the AZ CLI team as well regarding this issue. https://github.com/Azure/azure-cli/issues/11190

I’ve been banging my head for a while to no avail, happy I stumbled on this GH Issue, I’m having the same problem. Eagerly awaiting the bug fix. Can confirm the workaround of specifying the format as the Linux command date -u -d "1 day" '+%FT%TZ' to the parameter --expiry in az storage container generate-sas works for me too, I’m using ADLS Gen2.

This is also a problem when using SAS tokens with containers and storage policies (no explicit times in the signature). For example, this does not work on OSX, but does work on Windows using v10.3.1 on both.

./azcopy list https://myaccount.blob.core.windows.net/my-container?si=policy&sv=2019-02-02&sr=c&sig=redacted

@mifydnu glad to hear you figured it out! We are really sorry for the inconvenience. We’ll try to schedule this work item as soon as possible.

Apologies for being a little slow on the uptake here, but exactly what are the work-arounds? I’ve tried modifying the expiry time stamp in every way I can imagine with no luck. The sas I generate works fine in az commands, so I don’t think that’s the problem. Is there a know good format for expiry to use in, say, az storage container generate-sas’, that would provide a valid (for azCopy) sig? Thanks!