azure-sdk-for-java: [BUG]downloadToFile method has NPE error on 12.20.1 or newer version of azure-storage-blob

Describe the bug

When using downloadToFile method, NPE happens in case of 12.20.1 or over version of azure-storage-blob But it works well using 12.20.0 or older version.

Suspected change is https://github.com/Azure/azure-sdk-for-java/commit/c096193caa0406c9a6086f8e31adfc824a57abe7#diff-e54b10e2121ed9d4e7ec0eacf29f1feca0ae11a1294ac2f6b9d56bee8897b47eL138

Exception or Stack Trace Error stack

Exception in thread "main" java.lang.NullPointerException
        at com.azure.storage.blob.implementation.util.ChunkedDownloadUtils.extractTotalBlobLength(ChunkedDownloadUtils.java:138)
        at com.azure.storage.blob.implementation.util.ChunkedDownloadUtils.lambda$downloadFirstChunk$0(ChunkedDownloadUtils.java:57)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
        at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.onNext(MonoSubscribeOn.java:146)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
        at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
        at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
        at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
        at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2666)
        at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2402)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:150)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:189)
        at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
        at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
        at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:172)
        at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:414)
        at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:654)
        at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onStateChange(DefaultPooledConnectionProvider.java:201)
        at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onStateChange(DefaultPooledConnectionProvider.java:457)
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:628)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1374)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
                at reactor.core.publisher.Mono.block(Mono.java:1742)
                at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:156)
                at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1134)
                at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1095)
                at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:1004)
                at com.keyvault.Downloader.main(Downloader.java:47)

To Reproduce Steps to reproduce the behavior:

pom.xml

 <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-blob</artifactId>
      <version>12.20.1</version>
    </dependency>

java

        BlobClient blobClient = new BlobClientBuilder()
                .endpoint(SasUrl)
                .blobName(blobName)            
                .buildClient();

        blobClient.downloadToFile(enryptedFilePath, true);

Code Snippet Add the code snippet that causes the issue.

Expected behavior Downloaded normally

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

OS: [e.g. iOS] WSL2
IDE: [e.g. IntelliJ] VSCODE
Library/Libraries: com.azure:azure-storage-blob:12.20.1
Java version: [e.g. 8] 11
App Server/Environment: N/A
Frameworks: N/A

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn’t provide solution for the problem, please provide:

verbose dependency tree (mvn dependency:tree -Dverbose)
exception message, full stack trace, and any available logs

Additional context 12.20.0 version is working fine. From 12.20.1 to the latest version has NPE happening at com.azure.storage.blob.implementation.util.ChunkedDownloadUtils.extractTotalBlobLength

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[ V] Bug Description Added
[ V] Repro Steps Added
[ V] Setup information Added

About this issue

Original URL
State: closed
Created a year ago
Comments: 16 (8 by maintainers)

Most upvoted comments

@alzimmermsft Yeah. bom works perfectly. Recommends using bom dependency to the official document Thank you much!

HakjunMIN on May 16, 2023

Hi @HakjunMIN, looking into this error a bit more I believe you’re running into a dependency conflict issue. Based on the Azure SDK dependencies you’re using:

<dependencies>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-identity</artifactId>
    <version>1.3.5</version>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-security-keyvault-keys</artifactId>
    <version>4.4.0</version>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-core</artifactId>
    <version>1.38.0</version>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-security-keyvault-secrets</artifactId>
    <version>4.2.0</version>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.20.1</version>
    <!-- 12.20.1 or over notworking
    https://github.com/Azure/azure-sdk-for-java/commit/c096193caa0406c9a6086f8e31adfc824a57abe7#diff-e54b10e2121ed9d4e7ec0eacf29f1feca0ae11a1294ac2f6b9d56bee8897b47eL138
       -->
  </dependency>
</dependencies>

There will be conflicts with the version of azure-core-http-netty being resolved. azure-identity 1.3.5 uses azure-core-http-netty 1.10.2, azure-security-keyvault-keys 4.4.0 uses azure-core-http-netty 1.11.8, azure-security-keyvault-secrets 4.2.0 uses azure-core-http-netty 1.5.4, and azure-storage-blob 12.20.1 uses azure-core-http-netty 1.12.7, these are all different minor versions meaning there are API changes which newer versions of the SDKs (in this case azure-storage-blob) may depend on. My recommendation would be using our azure-sdk-bom to ensure that the versions of the SDKs being used are compatible. If you want to explicitly used azure-storage-blob 12.20.1 I’d recommend making this change to your dependencies (I’ve omitted non-Azure SDK dependencies):

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-sdk-bom</artifactId>
      <version>1.2.8</version> <!-- Use azure-storage-blob 12.20.1 -->
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-identity</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-security-keyvault-keys</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-core</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-security-keyvault-secrets</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
  </dependency>
</dependencies>

Or if you want to use the latest versions use azure-sdk-bom 1.2.12. In the long run this will simplify updating your Azure SDK dependencies to just updating the version of azure-sdk-bom being used, which we release every month at the time of writing this comment.

alzimmermsft on May 11, 2023

@ibrahimrabab Would you try one more time? just extended auth. https://sagmkt.blob.core.windows.net/test?sp=racwd&st=2023-05-02T02:00:30Z&se=2023-05-31T10:00:30Z&spr=https&sv=2022-11-02&sr=c&sig=NBl61nPX7k1F6SGO%2Bykma4i2KozKIQHygIm8igHymJM%3D

HakjunMIN on May 2, 2023