aws-sdk-go-v2: [SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules released on or after 11/15/23 are incompatible against previous runtimes (and vice versa)

Describe the bug

I am experiencing a not found, ResolveEndpointV2 error. This error occurs during the process of marshaling a JSON object and passing it to the SQS SendMessage method.

b, err := json.Marshal(*data)
if err != nil {
  return err
}

input := &sqs.SendMessageInput{
    MessageBody: aws.String(string(b)),
    QueueUrl:    aws.String(url),
}

if _, err := n.client.SendMessage(ctx, input); err != nil {
    return err
}

Expected Behavior

The SendMessage method is expected to execute without errors.

Current Behavior

When executing the SendMessage method, the following error occurs:

not found, ResolveEndpointV2

Reproduction Steps

  1. Create a JSON payload similar to the one shown below.
  2. Use the AWS SDK for Go to create an SQS client.
  3. Use the SendMessage method of the SQS client to send the JSON payload to a specified queue.
  4. Observe the ResolveEndpointV2 error during the execution of the SendMessage call.
{
   "target_id": "example-id",
   "target_body": "Example content",
   "target_type": 0,
   "event_type": 0,
   "deep_link_url": "exampleapp:///threads/example-id",
   "image_url": "https://example.com/icon.png",
   "tokens": null
}

Possible Solution

No response

Additional Information/Context

The issue does not occur with previous versions of the SDK, and reverting to an earlier version resolves the issue.

github.com/aws/aws-sdk-go v1.47.11
github.com/aws/aws-sdk-go-v2 v1.22.2
github.com/aws/aws-sdk-go-v2/config v1.25.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.42.1
github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89

AWS Go SDK V2 Module Versions Used

module cpf-api-service

go 1.21

require (
	github.com/aws/aws-sdk-go v1.47.12
	github.com/aws/aws-sdk-go-v2 v1.23.0
	github.com/aws/aws-sdk-go-v2/config v1.25.1
	github.com/aws/aws-sdk-go-v2/service/s3 v1.42.2
	github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89
	github.com/coreos/go-oidc/v3 v3.7.0
	github.com/danielgtaylor/huma v1.14.2
	github.com/gavv/httpexpect/v2 v2.16.0
	github.com/go-redis/redis/v8 v8.11.5
	github.com/go-sql-driver/mysql v1.7.1
	github.com/google/uuid v1.4.0
	github.com/google/wire v0.5.0
	github.com/kelseyhightower/envconfig v1.4.0
	github.com/lestrrat-go/jwx v1.2.26
	github.com/mitchellh/mapstructure v1.5.0
	github.com/pkg/errors v0.9.1
	github.com/uptrace/bun v1.1.16
	github.com/uptrace/bun/dbfixture v1.1.16
	github.com/uptrace/bun/dialect/mysqldialect v1.1.16
	github.com/uptrace/bun/extra/bundebug v1.1.16
	github.com/urfave/cli/v2 v2.25.7
	go.uber.org/zap v1.26.0
	goa.design/goa/v3 v3.14.0
	goa.design/plugins/v3 v3.14.0
	golang.org/x/oauth2 v0.14.0
	google.golang.org/api v0.150.0
	gopkg.in/DataDog/dd-trace-go.v1 v1.57.0
)

require (
	cloud.google.com/go v0.110.8 // indirect
	cloud.google.com/go/compute v1.23.1 // indirect
	cloud.google.com/go/firestore v1.13.0 // indirect
	cloud.google.com/go/iam v1.1.3 // indirect
	cloud.google.com/go/storage v1.30.1 // indirect
	firebase.google.com/go/v4 v4.12.1
	github.com/MicahParks/keyfunc v1.9.0 // indirect
	golang.org/x/text v0.14.0
	gopkg.in/yaml.v2 v2.4.0
)

require (
	github.com/avast/retry-go v3.0.0+incompatible
	github.com/aws/smithy-go v1.17.0
	github.com/google/go-cmp v0.6.0
	github.com/hololive/cpf-push-notification-service/go v0.0.0-20230306103827-2389f7bbb115
	github.com/nicksnyder/go-i18n/v2 v2.2.2
	github.com/rivo/uniseg v0.4.4
	github.com/spf13/cobra v1.8.0
	github.com/stretchr/testify v1.8.4
	golang.org/x/exp v0.0.0-20230321023759-10a507213a29
)

require (
	cloud.google.com/go/compute/metadata v0.2.3 // indirect
	cloud.google.com/go/longrunning v0.5.2 // indirect
	github.com/AnatolyRugalev/goregen v0.1.0 // indirect
	github.com/DataDog/appsec-internal-go v1.0.0 // indirect
	github.com/DataDog/datadog-agent/pkg/obfuscate v0.48.0 // indirect
	github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.48.1 // indirect
	github.com/DataDog/datadog-go/v5 v5.3.0 // indirect
	github.com/DataDog/go-libddwaf v1.5.0 // indirect
	github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
	github.com/DataDog/gostackparse v0.7.0 // indirect
	github.com/DataDog/sketches-go v1.4.2 // indirect
	github.com/Microsoft/go-winio v0.6.1 // indirect
	github.com/TylerBrock/colorjson v0.0.0-20200706003622-8a50f05110d2 // indirect
	github.com/ajg/form v1.5.1 // indirect
	github.com/andybalholm/brotli v1.0.6 // indirect
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.16.1 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.4 // indirect
	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.38 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.3 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.3 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.7.0 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.17.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.19.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.25.2 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/dimfeld/httppath v0.0.0-20170720192232-ee938bf73598 // indirect
	github.com/dustin/go-humanize v1.0.1 // indirect
	github.com/ebitengine/purego v0.5.0-alpha.1 // indirect
	github.com/fatih/color v1.15.0 // indirect
	github.com/fatih/structs v1.1.0 // indirect
	github.com/go-chi/chi/v5 v5.0.10 // indirect
	github.com/go-jose/go-jose/v3 v3.0.0 // indirect
	github.com/gobwas/glob v0.2.3 // indirect
	github.com/goccy/go-json v0.10.2 // indirect
	github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/go-querystring v1.1.0 // indirect
	github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b // indirect
	github.com/google/s2a-go v0.1.7 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
	github.com/gorilla/websocket v1.5.0 // indirect
	github.com/imkira/go-interpol v1.1.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jinzhu/inflection v1.0.0 // indirect
	github.com/jmespath/go-jmespath v0.4.0 // indirect
	github.com/klauspost/compress v1.17.1 // indirect
	github.com/lestrrat-go/backoff/v2 v2.0.8 // indirect
	github.com/lestrrat-go/blackmagic v1.0.1 // indirect
	github.com/lestrrat-go/httpcc v1.0.1 // indirect
	github.com/lestrrat-go/iter v1.0.2 // indirect
	github.com/lestrrat-go/option v1.0.1 // indirect
	github.com/manveru/faker v0.0.0-20171103152722-9fbc68a78c4d // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/outcaste-io/ristretto v0.2.3 // indirect
	github.com/philhofer/fwd v1.1.2 // indirect
	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
	github.com/richardartoul/molecule v1.0.1-0.20221107223329-32cfee06a052 // indirect
	github.com/russross/blackfriday/v2 v2.1.0 // indirect
	github.com/sanity-io/litter v1.5.5 // indirect
	github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
	github.com/sergi/go-diff v1.3.1 // indirect
	github.com/spaolacci/murmur3 v1.1.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/tinylib/msgp v1.1.8 // indirect
	github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
	github.com/valyala/bytebufferpool v1.0.0 // indirect
	github.com/valyala/fasthttp v1.50.0 // indirect
	github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
	github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
	github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
	github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
	github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 // indirect
	github.com/yudai/gojsondiff v1.0.0 // indirect
	github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.uber.org/atomic v1.11.0 // indirect
	go.uber.org/multierr v1.10.0 // indirect
	go4.org/intern v0.0.0-20230525184215-6c62f75575cb // indirect
	go4.org/unsafe/assume-no-moving-gc v0.0.0-20230525183740-e7c30c78aeb2 // indirect
	golang.org/x/crypto v0.15.0 // indirect
	golang.org/x/mod v0.13.0 // indirect
	golang.org/x/net v0.18.0 // indirect
	golang.org/x/sync v0.5.0 // indirect
	golang.org/x/sys v0.14.0 // indirect
	golang.org/x/time v0.3.0 // indirect
	golang.org/x/tools v0.14.0 // indirect
	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
	google.golang.org/appengine v1.6.8 // indirect
	google.golang.org/appengine/v2 v2.0.2 // indirect
	google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
	google.golang.org/grpc v1.59.0 // indirect
	google.golang.org/protobuf v1.31.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	inet.af/netaddr v0.0.0-20230525184311-b8eac61e914a // indirect
	moul.io/http2curl/v2 v2.3.0 // indirect
)

Compiler and Version used

go1.21.4 linux/arm64

Operating System and version

Debian GNU/Linux 12 (bookworm)

About this issue

  • Original URL
  • State: closed
  • Created 7 months ago
  • Reactions: 37
  • Comments: 28 (5 by maintainers)

Commits related to this issue

Most upvoted comments

To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue–

Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday’s release, that should resolve the issue. Please chime in with the results if you can.

It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).

For example, in the case of @ucirello’s package:

This combination of versions is broken:

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday, expects ResolveEndpointV2 to be in Finalize
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.1 // released before yesterday, ResolveEndpointV2 is in serialize
)

This combination is valid (I have verified that your tests under v2/ pass on my system after upgrading):

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.2 // released yesterday, ResolveEndpointV2 is now in Finalize
)

More detailed explanation to follow.

+25
lucix-aws on Nov 16, 2023

Glad to hear people are getting unblocked, we greatly appreciate everybody chiming into the thread. I’m going to leave this open at the top of the list for a while for visibility.

As for a post-mortem, this is clearly an unfortunate side-effect of the breaking change that went in on 11/15. In reality I suspect we have a host of undocumented compatibility breaks across our release history. Something as simple as an API added in one module to be consumed by another technically constitutes a break in this way, #2372 is an example of this. Even though our release tooling handles updating our own module dependencies, people are free to pin to their own versions and run into breaks like this.

The magnitude of this break is obviously much larger, but I can assert with confidence that changes on this level will be few and far between. The underlying change here was to reorder several key steps of the SDK operation lifecycle in order to align with the Smithy client reference architecture and enable advanced authentication behavior (as you may have guessed from the ubiquitous error message, endpoint resolution was one of those steps). With the way that the operation middleware API works (inserting steps into the operation can be keyed off of an existing slotted phase) it rendered anything pre-11/15 fundamentally broken against anything afterwards.

A true semantic versioning system would help mitigate this if not prevent it outright, but that’s not something we have in place at this time. Moreover that generally doesn’t align with our versioning strategy in the SDKs org where we tend to bake the “major version” into the repository name itself e.g. aws-sdk-go-v2, aws-sdk-java-v2. It’s something we’re considering moving forward, though, and this is obviously a strong supporting data point for that argument.

+14
lucix-aws on Nov 17, 2023

We’ve also noticed this bug in many of our modules in our dev environments since the sdk update, thank you for raising this issue.

Edit: And it’s not just affecting sqs. Many services seem to be returning this error, in particular on list/get operations which we’ve encountered in our integration tests. Our modules which worked fine before are failing for example when testing if specific Codebuild projects exist using codebuild.BatchGetProjects, returning the “ResolveEndpointV2” error.

+8
KaeDig on Nov 16, 2023

Command to update all AWS SDK dependencies in one go:

$ go get -u "github.com/aws/aws-sdk-go-v2/..."
+7
droberts-sea on Dec 19, 2023

I was encountering the not found, ResolveEndpointV2 error while trying to upload a file to S3. Updating the S3 dependency with go get github.com/aws/aws-sdk-go-v2/service/s3@latest resolved the issue.

+7
chrisclapham on Nov 29, 2023

We are also facing same issue while initialising the aws.NewCredentialsCache(creds) and trying to retrieve the creds from the cache.

role := "arn:aws:iam::1222233333:role/sample-assume-role"
creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
		aro.RoleSessionName = "RoleSession"
		aro.RoleARN =  role
	})

credCache := aws.NewCredentialsCache(creds)
credentials, err := credCache.Retrieve(ctx)
if err != nil {
   // getting error as "failed to refresh cached credentials, not found, ResolveEndpointV2"

    return aws.Credentials{}, err 
}
+6
DeeptimanQlik on Nov 16, 2023

I hit this issue and got past it by upgrading the github.com/aws/aws-sdk-go-v2 packages to the latest version.

+5
jkakar on Nov 16, 2023

To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue–

Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday’s release, that should resolve the issue. Please chime in with the results if you can.

It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).

For example, in the case of @ucirello’s package:

This combination of versions is broken:

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday, expects ResolveEndpointV2 to be in Finalize
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.1 // released before yesterday, ResolveEndpointV2 is in serialize
)

This combination is valid (I have verified that your tests under v2/ pass on my system after upgrading):

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.2 // released yesterday, ResolveEndpointV2 is now in Finalize
)

More detailed explanation to follow.

Thanks, it works for me with aws.NewCredentialsCache

+5
sawtheshib on Nov 16, 2023

@lucix-aws Upgrading all the pending chore updates into one branch solves the issues for me.

go get github.com/aws/aws-sdk-go-v2@v1.23.0
go get github.com/aws/aws-sdk-go-v2/service/kms@v1.26.2
go get github.com/aws/aws-sdk-go-v2/credentials@v1.16.1
go get github.com/aws/aws-sdk-go-v2/config@1.25.1
+3
DeeptimanQlik on Nov 16, 2023

I just read the Release Notes. I think this breaking change is likely the cause.

Feature: BREAKING CHANGE: V2 endpoint resolution middleware has changed steps from Serialize to Finalize. Middleware that indexes off of this field will need to be updated accordingly.

+3
khorii-cover on Nov 16, 2023

@lucix-aws Sorry for the late reply. It seems that updating the SQS and config has resolved the issue. In our project, we are auto-updating several modules, but one of them was not well-managed, and the issue was caused by one of the modules being outdated.

+2
khorii-cover on Nov 17, 2023

We are also facing same issue while initialising the aws.NewCredentialsCache(creds) and trying to retrieve the creds from the cache.

role := "arn:aws:iam::1222233333:role/sample-assume-role"
creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
		aro.RoleSessionName = "RoleSession"
		aro.RoleARN =  role
	})

credCache := aws.NewCredentialsCache(creds)
credentials, err := credCache.Retrieve(ctx)
if err != nil {
   // getting error as "failed to refresh cached credentials, not found, ResolveEndpointV2"

    return aws.Credentials{}, err 
}

I think it’s because the same addProtocolFinalizerMiddlewares function is being called internally.

https://github.com/aws/aws-sdk-go-v2/blob/435199fc01ab47020ab36dab07d8115e20687f73/service/sts/api_op_AssumeRole.go#L87

+2
khorii-cover on Nov 16, 2023

@lucix-aws the new version of config seems to have fixed the error. Thanks!

go: upgraded github.com/aws/aws-sdk-go-v2/config v1.25.1 => v1.25.2
+1
koreyGambill on Nov 16, 2023

I think this line might possibly be where the error is occurring.

https://github.com/aws/smithy-go/blob/main/middleware/ordered_group.go#L217

+1
khorii-cover on Nov 16, 2023
Read more comments on GitHub
← aws-sdk-go-v2: ParseInt Error with role duration seconds key
aws-sdk-java-v2: closeEventLoopUninterruptibly Hanging on WebFlux Netty →