aws-sdk-kotlin: Binary incompatibility with ktor-io 2.x results in NoSuchMethodError exceptions at runtime

Describe the bug

When using response.body?.toByteArray(), I get the following errors :

23:47:40.748 [DefaultDispatcher-worker-5] DEBUG httpTraceMiddleware - service: S3; operation: GetObject; - HttpResponse: 200: OK
23:47:40.767 [DefaultDispatcher-worker-5] ERROR ktor.application - Websocket handler failed
java.lang.NoSuchMethodError: 'void io.ktor.utils.io.bits.Memory.copyTo-iAfECsU(java.nio.ByteBuffer, java.nio.ByteBuffer, long, long, long)'
	at aws.smithy.kotlin.runtime.io.SdkByteBufferKt.readFully-aPcrCvc(SdkByteBuffer.kt:377)
	at aws.sdk.kotlin.runtime.http.engine.crt.SegmentKt.copyTo(Segment.kt:21)
	at aws.sdk.kotlin.runtime.http.engine.crt.AbstractBufferedReadChannel.readAsMuchAsPossible(AbstractBufferedReadChannel.kt:113)
	at aws.sdk.kotlin.runtime.http.engine.crt.AbstractBufferedReadChannel.readRemaining$suspendImpl(AbstractBufferedReadChannel.kt:97)
	at aws.sdk.kotlin.runtime.http.engine.crt.AbstractBufferedReadChannel.readRemaining(AbstractBufferedReadChannel.kt)
	at aws.smithy.kotlin.runtime.io.SdkByteReadChannel$DefaultImpls.readRemaining$default(SdkByteReadChannelJVM.kt:35)
	at aws.smithy.kotlin.runtime.content.ByteStreamKt.consumeStream(ByteStream.kt:67)
	at aws.smithy.kotlin.runtime.content.ByteStreamKt.toByteArray(ByteStream.kt:83)

Expected behavior

We are able to read the response body.

Current behavior

Trying to read the response body throws an exception java.lang.NoSuchMethodError

Steps to Reproduce

Here is the code to reproduce (taken from the doc pretty much as-is)

import aws.sdk.kotlin.services.s3.S3Client
import aws.sdk.kotlin.services.s3.model.GetObjectRequest
import aws.smithy.kotlin.runtime.content.toByteArray  
import kotlinx.coroutines.async

private fun downloadImageFromS3(imageLocation: String) =
    async {
        val request = createS3GetObjectRequest(imageLocation)
	var imageBytes: ByteArray? = null

	S3Client(s3Configuration).use { s3 ->
		s3.getObject(request) { response ->
			imageBytes = response.body?.toByteArray()
		}
	}

	imageBytes
}

private fun createS3GetObjectRequest(imageLocation: String) =
	GetObjectRequest {
		key = imageLocation
		bucket = s3BucketName
	}

Possible Solution

No response

Context

I am running Ktor 2.0.0-beta-1. The error still occurs after deleting all the libraries downloaded, deleting gradle cache then re-downloading them via gradle sync.
Here is a screenshot of the dependencies that the S3 lib downloaded
image

AWS Kotlin SDK version used

0.9.5-beta

Platform (JVM/JS/Native)

JVM 11

Operating System and version

Ubuntu 20.04.3 LTS

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 24 (3 by maintainers)

Commits related to this issue

Most upvoted comments

This will be available in the next release (0.16.0-beta) today or tomorrow hopefully.

Please fix this soon, my application relies on it. I’m too far in on my project to revert back to last ktor version

Not at present no. The fix requires us to either upgrade (which will reverse the issue and cause binary incompatibility with 1.x users of ktor) OR shade ktor-io:1.x into the SDK.

Our current plan, as Ian noted, is to look into upgrading to 2.0 in the near term and see what feedback we get to determine if we need to do anything further.

Yes Ktor 2.0 has been officially released and we’re now evaluating moving the SDK’s underlying version from 1.x to 2.0. Until that work is complete, you will likely still see errors when mixing the AWS SDK for Kotlin and Ktor 2.x.

The 0.16.0 release which upgrades to ktor 2.0.1 has made it’s way to maven central. This build also includes a changeover to KMP builds/artifacts. If you are using Gradle the Kotlin plugin (whether you use multiplatform or not) should take care of this for you, if not you may need to depend on the -jvm version (e.g. s3-jvm)of specific services. We of course still only support the JVM target at this time though.

NOTE: We accidentally published this without the -beta qualifier due to a miss in some new publishing infrastructure. Future dev preview releases will once again use this qualifier.

Is there an ETA on this effort? I just upgraded to Ktor 2 without realizing the consequences. The aws-sdk-kotlin isn’t 100% vital for my application. My options are wait for a Ktor 2.x compatible version, or switch back to the AWS Java SDK. If this is fixed soon, my preference is leave things broken and not get notifications for a few days.

Agreed, we’ll leave this issue open for tracking, performing the upgrade/shading, and documenting the final outcomes.

@aajtodd I agree with you, it’s important to consider the impact of the fix on 1.x users.
It’s a good thing you notified JB about it as this seems like an issue that can affect a lot of libraries like you said.

We will keep an eye for any update from your part on this as this blocks us from downloading S3 objects for now.

Thanks