aws-sdk-go-v2: [SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules released on or after 11/15/23 are incompatible against previous runtimes (and vice versa)
Describe the bug
I am experiencing a not found, ResolveEndpointV2
error.
This error occurs during the process of marshaling a JSON object and passing it to the SQS SendMessage method.
b, err := json.Marshal(*data)
if err != nil {
return err
}
input := &sqs.SendMessageInput{
MessageBody: aws.String(string(b)),
QueueUrl: aws.String(url),
}
if _, err := n.client.SendMessage(ctx, input); err != nil {
return err
}
Expected Behavior
The SendMessage method is expected to execute without errors.
Current Behavior
When executing the SendMessage method, the following error occurs:
not found, ResolveEndpointV2
Reproduction Steps
- Create a JSON payload similar to the one shown below.
- Use the AWS SDK for Go to create an SQS client.
- Use the SendMessage method of the SQS client to send the JSON payload to a specified queue.
- Observe the ResolveEndpointV2 error during the execution of the SendMessage call.
{
"target_id": "example-id",
"target_body": "Example content",
"target_type": 0,
"event_type": 0,
"deep_link_url": "exampleapp:///threads/example-id",
"image_url": "https://example.com/icon.png",
"tokens": null
}
Possible Solution
No response
Additional Information/Context
The issue does not occur with previous versions of the SDK, and reverting to an earlier version resolves the issue.
github.com/aws/aws-sdk-go v1.47.11
github.com/aws/aws-sdk-go-v2 v1.22.2
github.com/aws/aws-sdk-go-v2/config v1.25.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.42.1
github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89
AWS Go SDK V2 Module Versions Used
module cpf-api-service
go 1.21
require (
github.com/aws/aws-sdk-go v1.47.12
github.com/aws/aws-sdk-go-v2 v1.23.0
github.com/aws/aws-sdk-go-v2/config v1.25.1
github.com/aws/aws-sdk-go-v2/service/s3 v1.42.2
github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89
github.com/coreos/go-oidc/v3 v3.7.0
github.com/danielgtaylor/huma v1.14.2
github.com/gavv/httpexpect/v2 v2.16.0
github.com/go-redis/redis/v8 v8.11.5
github.com/go-sql-driver/mysql v1.7.1
github.com/google/uuid v1.4.0
github.com/google/wire v0.5.0
github.com/kelseyhightower/envconfig v1.4.0
github.com/lestrrat-go/jwx v1.2.26
github.com/mitchellh/mapstructure v1.5.0
github.com/pkg/errors v0.9.1
github.com/uptrace/bun v1.1.16
github.com/uptrace/bun/dbfixture v1.1.16
github.com/uptrace/bun/dialect/mysqldialect v1.1.16
github.com/uptrace/bun/extra/bundebug v1.1.16
github.com/urfave/cli/v2 v2.25.7
go.uber.org/zap v1.26.0
goa.design/goa/v3 v3.14.0
goa.design/plugins/v3 v3.14.0
golang.org/x/oauth2 v0.14.0
google.golang.org/api v0.150.0
gopkg.in/DataDog/dd-trace-go.v1 v1.57.0
)
require (
cloud.google.com/go v0.110.8 // indirect
cloud.google.com/go/compute v1.23.1 // indirect
cloud.google.com/go/firestore v1.13.0 // indirect
cloud.google.com/go/iam v1.1.3 // indirect
cloud.google.com/go/storage v1.30.1 // indirect
firebase.google.com/go/v4 v4.12.1
github.com/MicahParks/keyfunc v1.9.0 // indirect
golang.org/x/text v0.14.0
gopkg.in/yaml.v2 v2.4.0
)
require (
github.com/avast/retry-go v3.0.0+incompatible
github.com/aws/smithy-go v1.17.0
github.com/google/go-cmp v0.6.0
github.com/hololive/cpf-push-notification-service/go v0.0.0-20230306103827-2389f7bbb115
github.com/nicksnyder/go-i18n/v2 v2.2.2
github.com/rivo/uniseg v0.4.4
github.com/spf13/cobra v1.8.0
github.com/stretchr/testify v1.8.4
golang.org/x/exp v0.0.0-20230321023759-10a507213a29
)
require (
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/longrunning v0.5.2 // indirect
github.com/AnatolyRugalev/goregen v0.1.0 // indirect
github.com/DataDog/appsec-internal-go v1.0.0 // indirect
github.com/DataDog/datadog-agent/pkg/obfuscate v0.48.0 // indirect
github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.48.1 // indirect
github.com/DataDog/datadog-go/v5 v5.3.0 // indirect
github.com/DataDog/go-libddwaf v1.5.0 // indirect
github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
github.com/DataDog/gostackparse v0.7.0 // indirect
github.com/DataDog/sketches-go v1.4.2 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/TylerBrock/colorjson v0.0.0-20200706003622-8a50f05110d2 // indirect
github.com/ajg/form v1.5.1 // indirect
github.com/andybalholm/brotli v1.0.6 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.16.1 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.4 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.38 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.3 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.7.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.17.2 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.19.2 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.25.2 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dimfeld/httppath v0.0.0-20170720192232-ee938bf73598 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ebitengine/purego v0.5.0-alpha.1 // indirect
github.com/fatih/color v1.15.0 // indirect
github.com/fatih/structs v1.1.0 // indirect
github.com/go-chi/chi/v5 v5.0.10 // indirect
github.com/go-jose/go-jose/v3 v3.0.0 // indirect
github.com/gobwas/glob v0.2.3 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/imkira/go-interpol v1.1.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/klauspost/compress v1.17.1 // indirect
github.com/lestrrat-go/backoff/v2 v2.0.8 // indirect
github.com/lestrrat-go/blackmagic v1.0.1 // indirect
github.com/lestrrat-go/httpcc v1.0.1 // indirect
github.com/lestrrat-go/iter v1.0.2 // indirect
github.com/lestrrat-go/option v1.0.1 // indirect
github.com/manveru/faker v0.0.0-20171103152722-9fbc68a78c4d // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/outcaste-io/ristretto v0.2.3 // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/richardartoul/molecule v1.0.1-0.20221107223329-32cfee06a052 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sanity-io/litter v1.5.5 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
github.com/sergi/go-diff v1.3.1 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/tinylib/msgp v1.1.8 // indirect
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.50.0 // indirect
github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 // indirect
github.com/yudai/gojsondiff v1.0.0 // indirect
github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // indirect
go.opencensus.io v0.24.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.10.0 // indirect
go4.org/intern v0.0.0-20230525184215-6c62f75575cb // indirect
go4.org/unsafe/assume-no-moving-gc v0.0.0-20230525183740-e7c30c78aeb2 // indirect
golang.org/x/crypto v0.15.0 // indirect
golang.org/x/mod v0.13.0 // indirect
golang.org/x/net v0.18.0 // indirect
golang.org/x/sync v0.5.0 // indirect
golang.org/x/sys v0.14.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.14.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/appengine/v2 v2.0.2 // indirect
google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
google.golang.org/grpc v1.59.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
inet.af/netaddr v0.0.0-20230525184311-b8eac61e914a // indirect
moul.io/http2curl/v2 v2.3.0 // indirect
)
Compiler and Version used
go1.21.4 linux/arm64
Operating System and version
Debian GNU/Linux 12 (bookworm)
About this issue
- Original URL
- State: closed
- Created 7 months ago
- Reactions: 37
- Comments: 28 (5 by maintainers)
Commits related to this issue
- all: upgrade deps for aws/aws-sdk-go-v2#2370 — committed to FiloSottile/litetlog by FiloSottile 7 months ago
- chore(deps): upgrade aws sdkv2 for aws/aws-sdk-go-v2#2370 — committed to wolfeidau/s3iofs by wolfeidau 7 months ago
- Merge pull request #22 from wolfeidau/fix_upgrade_aws_sdkv2 chore(deps): upgrade aws sdkv2 for aws/aws-sdk-go-v2#2370 — committed to wolfeidau/s3iofs by wolfeidau 7 months ago
- use latest SDK refs https://github.com/aws/aws-sdk-go-v2/issues/2370 — committed to fujiwara/lambroll by fujiwara 7 months ago
- chore: bump all aws-sdk-go-v2 dependencies Please see https://github.com/aws/aws-sdk-go-v2/issues/2370 — committed to mdelapenya/testcontainers-go by mdelapenya 7 months ago
- chore(deps): bump github.com/aws/aws-sdk-go and github.com/aws/aws-sdk-go-v2 in /modules/localstack (#1953) * chore(deps): bump github.com/aws/aws-sdk-go-v2 in /modules/localstack Bumps [github.co... — committed to testcontainers/testcontainers-go by mdelapenya 7 months ago
- Bump other AWS SDK dependencies per GitHub issue (https://github.com/aws/aws-sdk-go-v2/issues/2370) — committed to evergreen-ci/evergreen by Kimchelly 7 months ago
- fix: also update github.com/awslabs/amazon-ecr-credential-helper to resolve issues with AWS ECR authentication (resolves #2882) As mentioned in https://github.com/aws/aws-sdk-go-v2/issues/2370, AWS S... — committed to pdecat/kaniko by pdecat 7 months ago
- fix: also update github.com/awslabs/amazon-ecr-credential-helper to resolve issues with AWS ECR authentication As mentioned in https://github.com/aws/aws-sdk-go-v2/issues/2370, AWS SDK for Go v2 rele... — committed to pdecat/kaniko by pdecat 7 months ago
- fix: also update github.com/awslabs/amazon-ecr-credential-helper to resolve issues with AWS ECR authentication (#2908) As mentioned in https://github.com/aws/aws-sdk-go-v2/issues/2370, AWS SDK for G... — committed to GoogleContainerTools/kaniko by pdecat 7 months ago
To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue–
Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday’s release, that should resolve the issue. Please chime in with the results if you can.
It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).
For example, in the case of @ucirello’s package:
This combination of versions is broken:
This combination is valid (I have verified that your tests under
v2/
pass on my system after upgrading):More detailed explanation to follow.
Glad to hear people are getting unblocked, we greatly appreciate everybody chiming into the thread. I’m going to leave this open at the top of the list for a while for visibility.
As for a post-mortem, this is clearly an unfortunate side-effect of the breaking change that went in on 11/15. In reality I suspect we have a host of undocumented compatibility breaks across our release history. Something as simple as an API added in one module to be consumed by another technically constitutes a break in this way, #2372 is an example of this. Even though our release tooling handles updating our own module dependencies, people are free to pin to their own versions and run into breaks like this.
The magnitude of this break is obviously much larger, but I can assert with confidence that changes on this level will be few and far between. The underlying change here was to reorder several key steps of the SDK operation lifecycle in order to align with the Smithy client reference architecture and enable advanced authentication behavior (as you may have guessed from the ubiquitous error message, endpoint resolution was one of those steps). With the way that the operation middleware API works (inserting steps into the operation can be keyed off of an existing slotted phase) it rendered anything pre-11/15 fundamentally broken against anything afterwards.
A true semantic versioning system would help mitigate this if not prevent it outright, but that’s not something we have in place at this time. Moreover that generally doesn’t align with our versioning strategy in the SDKs org where we tend to bake the “major version” into the repository name itself e.g.
aws-sdk-go-v2
,aws-sdk-java-v2
. It’s something we’re considering moving forward, though, and this is obviously a strong supporting data point for that argument.We’ve also noticed this bug in many of our modules in our dev environments since the sdk update, thank you for raising this issue.
Edit: And it’s not just affecting sqs. Many services seem to be returning this error, in particular on list/get operations which we’ve encountered in our integration tests. Our modules which worked fine before are failing for example when testing if specific Codebuild projects exist using codebuild.BatchGetProjects, returning the “ResolveEndpointV2” error.
Command to update all AWS SDK dependencies in one go:
I was encountering the
not found, ResolveEndpointV2
error while trying to upload a file to S3. Updating the S3 dependency withgo get github.com/aws/aws-sdk-go-v2/service/s3@latest
resolved the issue.We are also facing same issue while initialising the
aws.NewCredentialsCache(creds)
and trying to retrieve the creds from the cache.I hit this issue and got past it by upgrading the
github.com/aws/aws-sdk-go-v2
packages to the latest version.Thanks, it works for me with aws.NewCredentialsCache
@lucix-aws Upgrading all the pending chore updates into one branch solves the issues for me.
I just read the Release Notes. I think this breaking change is likely the cause.
@lucix-aws Sorry for the late reply. It seems that updating the SQS and config has resolved the issue. In our project, we are auto-updating several modules, but one of them was not well-managed, and the issue was caused by one of the modules being outdated.
I think it’s because the same
addProtocolFinalizerMiddlewares
function is being called internally.https://github.com/aws/aws-sdk-go-v2/blob/435199fc01ab47020ab36dab07d8115e20687f73/service/sts/api_op_AssumeRole.go#L87
@lucix-aws the new version of config seems to have fixed the error. Thanks!
I think this line might possibly be where the error is occurring.
https://github.com/aws/smithy-go/blob/main/middleware/ordered_group.go#L217