aws-sdk-java-v2: S3TransferManager SdkClientException: Failed to send the request: Retry cannot be attempted because the retry strategy has prevented
Describe the bug
Sometimes large files (1GB ) upload failed with SDKClientException “Failed to send the request: Retry cannot be attempted because the retry strategy has prevented”.
The S3 Transfer Manager is configured as:
this.transferManager =
S3TransferManager.builder()
.s3ClientConfiguration(
cfg ->
cfg.credentialsProvider(awsCredentialsProvider)
.region(Region.of(s3Region))
.targetThroughputInGbps(20.0)
.minimumPartSizeInBytes(5 * SizeConstant.MB))
.build();
I didn’t prevent the retry strategy. I’m wondering if this is prevented by default.
Expected Behavior
Files could be uploaded successfully.
Current Behavior
Exceptions thrown:
Caused by: java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the retry strategy has prevented the operation.
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:181)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Retry cannot be attempted because the retry strategy has prevented the operation.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at software.amazon.awssdk.transfer.s3.internal.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:93)
at software.amazon.awssdk.transfer.s3.internal.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:73)
at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:19)
Reproduction Steps
This issue may be caused by unstable network and is hard to reproduce.
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.17.191-PREVIEW
JDK version used
1.8.0_311
Operating System and version
macOS Monterey 12.5.1
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (3 by maintainers)
In C, we default to 10 Gbps, here. For CLI, we default to 5 Gbps here. I am not sure about Java, @zoewangg may know better.
It’s useful to set, like our test for best performance from c5n.18xlarge instance will set it 100. And for your use case in poor network, it’s probably better to set it to 1 or something.
@TingDaoK Is there any way we can improve the error message? The current message “Failed to send the request: Retry cannot be attempted because the retry strategy has prevented the operation.” does not provide much information. If CRT could return a specific error code for this case, we can add the guidance in the error message.