azure-sdk-for-go: Azure Blob: Unable to access Metadata when listing blobs

Bug Report

• import path of package in question, e.g. github.com/Azure/azure-sdk-for-go/sdk/storage/azblob

require github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.2.0

require ( github.com/Azure/azure-sdk-for-go/sdk/azcore v0.20.0 // indirect github.com/Azure/azure-sdk-for-go/sdk/internal v0.8.1 // indirect github.com/stretchr/objx v0.2.0 // indirect )

• output of go version go version go1.17.2 darwin/arm64

• What happened?

I’m trying to list metadata associated with blobs, but it appears to be incorrectly handled in the SDK code so it always comes back empty.

To reproduce, upload some blobs with custom metadata like so:

	resp, err := bc.Upload(ctx, data.Reader(), &azblob.UploadBlockBlobOptions{
		Metadata: map[string]string{"Kopiamtime": "1577968240000000000"},
	})

Now when listing the blobs there are 2 sub-issues, one blocking the other:

1. Incorrect handlng of ContainerListBlobFlatSegmentOptions.Include:

My first attempt was to try:

bucket.ListBlobsFlat(&azblob.ContainerListBlobFlatSegmentOptions{
		Prefix: &prefixStr,
		Include: []azblob.ListBlobsIncludeItem{azblob.ListBlobsIncludeItemMetadata},
	})

but the server never returns any metadata. I read through the code and I think there’s a bug generating the include query parameter and server silently ignores invalid value (bug in itself?), and to overcome it I have to add artificial square brackets, which makes it work partially - see below:

bucket.ListBlobsFlat(&azblob.ContainerListBlobFlatSegmentOptions{
		Prefix: &prefixStr,
		Include: []azblob.ListBlobsIncludeItem{"["+azblob.ListBlobsIncludeItemMetadata+"]"},
	})

1. Incorrect mapping of Metadata

After applying the workaround above, I get the following response from the server (there are 4 files in the bucket that match the prefix) - the response was captured using io.Copy(os.Stderr, resp.RawResponse.Body):

<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://kopiatesting.blob.core.windows.net/" ContainerName="kopia-testing">
    <Prefix>sastest-1639703094-ed1304df8b99da34-</Prefix>
    <Blobs>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-abcdbbf4f0507d054ed5a80a5b65086f602b</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93CEC42AE</Etag>
                <Content-Length>0</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>1B2M2Y8AsgTpgAmY7PhCfg==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-abff4585856ebf0748fd989e1dd623a8963d</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D28CE3A</Etag>
                <Content-Length>1000</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>8INdvman6QsODoZHNU0MCw==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-abgc3dca496d510f492c858a2df1eb824e62</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D3F6062</Etag>
                <Content-Length>10000</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>Gvp7eHd5/MlEV1brkD9viQ==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-kopia.repository</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D5640A2</Etag>
                <Content-Length>100</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>S3ixhZH4sRD+KIqBPwIHFw==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
        <Blob>
            <Name>sastest-1639703094-ed1304df8b99da34-zxce0e35630770c54668a8cfb4e414c6bf8f</Name>
            <Properties>
                <Creation-Time>Fri, 17 Dec 2021 01:04:55 GMT</Creation-Time>
                <Last-Modified>Fri, 17 Dec 2021 01:04:55 GMT</Last-Modified>
                <Etag>0x8D9C0F93D11C6F2</Etag>
                <Content-Length>1</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-CRC64 />
                <Content-MD5>VaVACK0bpYmqIQ0mKcHfQQ==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <AccessTier>Hot</AccessTier>
                <AccessTierInferred>true</AccessTierInferred>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>true</ServerEncrypted>
            </Properties>
            <Metadata>
                <Kopiamtime>1577968240000000000</Kopiamtime>
            </Metadata>
            <OrMetadata />
        </Blob>
    </Blobs>
    <NextMarker />
</EnumerationResults>

I was looking to retrieve metadata named Kopiamtime which shows up in the server response, but unfortunately cannot be read in Go code, because it.Metadata is always an empty map:

for _, it := range resp.Segment.BlobItems {
  fmt.Println(it.Metadata.AdditionalProperties)
}           

This always prints map[].

I’m not an expert in Golang XML handling, but I think this mapping is to blame, probably because of extra nesting:

https://github.com/Azure/azure-sdk-for-go/blob/a330caebda630563b60b5006a5485daa68d1f75d/sdk/storage/azblob/zz_generated_models.go#L381

Also notice how the server returns OrMetadata but the code is mapping ObjectReplicationMetadata which is also incorrect.

BTW. Thanks for the library. I’m trying to use it in https://github.com/kopia/kopia - all other things are working fine, except this one issue which is currently blocking.

Is there a workaround that can be applied here? I was thinking of manually parsing the XML for the time being until this issue is resolved.