aws-sdk-js-v3: EndpointFunctions[fn] is not a function

Checkboxes for prior research

Describe the bug

aws-sdk recently switched from @aws-sdk/util-endpoints to @smithy/util-endpoints, when calling client-lambda inside a container, Error: endpointFunctions[fn] is not a function will appear when calling lambda functions.

SDK version number

@aws-sdk/client-lambda@3.438

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v18.18.2

Reproduction Steps

import { LambdaClient, InvokeCommand, InvocationType } from '@aws-sdk/client-lambda'; // ES Modules import
.
.
.
const input = {
	FunctionName: `some-function`,
	InvocationType: InvocationType.RequestResponse,
	Payload: JSON.stringify({
        . . .
         }),
};

const command = new InvokeCommand(input);
const response = await client.send(command);

Observed Behavior

@smithy/util-endpoints was not able to find a function call endpointFunctions

Expected Behavior

@smithy/util-endpoints able to locate and call endpointFunctions

Possible Solution

roll back to “@aws-sdk/client-lambda”: “3.418.0”

Additional Information/Context

No response

About this issue

  • Original URL
  • State: closed
  • Created 8 months ago
  • Reactions: 21
  • Comments: 30 (3 by maintainers)

Commits related to this issue

Most upvoted comments

I can confirm that this issue arises when package version resolution results in multiple versions of @smithy/util-endpoints being used at runtime. The way @smithy/util-endpoints is currently architected, with a global custom endpoint function registry, makes it unsafe to have multiple copies in use simultaneously. This should be fixed in aws-sdk.

In the meantime, a workaround is to use overrides (npm), pnpm.overrides (pnpm), or resolutions (yarn) to force @smithy/util-endpoints to be resolved to a single version only.

+5
jfirebaugh on Jan 22, 2024

I’m seeing the same issue. I don’t believe this should be closed.

I reverted back to 3.4.28.0 and it seems to work again.

+5
seawatts on Nov 15, 2023

I got this issue too. Turned out I had one of my local modules called aws (guess what it’s responsible for 😁). So the line export * from "./aws"; in @aws-sdk/util-endpoints picked the wrong one. And the function which was not found was aws.partition. (Maybe it’s the side effect of bundling ncc).

The aws call I’m trying to make is getSignedUrl from @aws-sdk/s3-request-presigner and PutObjectCommand from @aws-sdk/client-s3

I don’t have the same export * from "./aws" in my project. However, when I debug the program, it fails here: https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-endpoints/src/utils/callFunction.ts#L16 The function that is trying to be passed in is fn = aws.partition, aws.isVirtualHostableS3Bucket, and aws.parseArn The available functions to call are

booleanEquals
getAttr
isSet
isValidHostLabel
not
parseURL
normalizedPath
stringEquals
substring
uriEncode

The total ruleset is defined here: https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/src/endpoint/ruleset.ts

This is where one of the conditions aws.partition is defined in that file https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/src/endpoint/ruleset.ts#L29

The rule condition is coming from many rules, here are a couple.

  • Partition does not support FIPS
  • Unrecognized hardware type: “Expected hardware type o or e but got {hardwareType}”
  • Invalid ARN: The outpost Id must only contain a-z, A-Z, 0-9 and -.

Going back in the history of that file, that condition has always been there. The smithy client must not be able to handle those functions, but the old utils must have been able to. I haven’t dove into the old utils that shipped with this package. I believe the change happened with this PR https://github.com/aws/aws-sdk-js-v3/pull/5390

It looks like the latest version of the @aws-sdk/client-s3 that works before that pull request was 3.437.0

+4
seawatts on Dec 14, 2023

I have the same issue here… when I upgrade my project from Next.js 13 to Next.js 14. Unfortunately, I am trying to reproduce it on a small project, but there, it works like a charm so far 😭

I have a reproducible test case ✅ 🚀

Same repo as earlier, but I’ve modified the setup to a pnpm monorepo https://github.com/janvorwerk/aws-s3-demo

  • Works like a charm in Next.js 13
  • Updating to Next.js 14 produces the same error!

All hints about multiple versions of @smithy/util-endpoints fall short in my case… I suppose the packaging induces multiple instances, but there is only one single version in my depenencies… pinning a version does not help!

+3
janvorwerk on Mar 20, 2024

I got this issue too. Turned out I had one of my local modules called aws (guess what it’s responsible for 😁). So the line export * from "./aws"; in @aws-sdk/util-endpoints picked the wrong one. And the function which was not found was aws.partition. (Maybe it’s the side effect of bundling ncc).

+3
ytimenkov on Nov 20, 2023

I have the same issue here… when I upgrade my project from Next.js 13 to Next.js 14. Unfortunately, I am trying to reproduce it on a small project, but there, it works like a charm so far 😭

I have a reproducible test case ✅ 🚀

Same repo as earlier, but I’ve modified the setup to a pnpm monorepo https://github.com/janvorwerk/aws-s3-demo

  • Works like a charm in Next.js 13
  • Updating to Next.js 14 produces the same error!

All hints about multiple versions of @smithy/util-endpoints fall short in my case… I suppose the packaging induces multiple instances, but there is only one single version in my depenencies… pinning a version does not help!

@janvorwerk We are also facing the same issue. We tried pinning the version, but no success.

+2
coronapl on Mar 20, 2024

I can confirm that this issue arises when package version resolution results in multiple versions of @smithy/util-endpoints being used at runtime. The way @smithy/util-endpoints is currently architected, with a global custom endpoint function registry, makes it unsafe to have multiple copies in use simultaneously. This should be fixed in aws-sdk.

In the meantime, a workaround is to use overrides (npm), pnpm.overrides (pnpm), or resolutions (yarn) to force @smithy/util-endpoints to be resolved to a single version only.

Pinning the package to 1.1.2 worked for me:

// package.json

"pnpm": {
    "overrides": {
      "@smithy/util-endpoints": "1.1.2"
    }
  },

pnpm why @smithy/util-endpoints

sst 2.40.3
└─┬ @aws-sdk/client-cloudformation 3.511.0
└─┬ @aws-sdk/client-sts 3.511.0
  └─┬ @aws-sdk/credential-provider-node 3.511.0 peer
    ├─┬ @aws-sdk/credential-provider-ini 3.511.0
    │ └─┬ @aws-sdk/credential-provider-sso 3.511.0
    │   ├─┬ @aws-sdk/client-sso 3.511.0
    │   │ ├─┬ @aws-sdk/middleware-user-agent 3.511.0
    │   │ │ └─┬ @aws-sdk/util-endpoints 3.511.0
    │   │ │   └── @smithy/util-endpoints 1.1.2
    │   │ ├─┬ @aws-sdk/util-endpoints 3.511.0
    │   │ │ └── @smithy/util-endpoints 1.1.2
    │   │ └── @smithy/util-endpoints 1.1.2
    │   └─┬ @aws-sdk/token-providers 3.511.0
    │     └─┬ @aws-sdk/client-sso-oidc 3.511.0
    │       ├─┬ @aws-sdk/middleware-user-agent 3.511.0
    │       │ └─┬ @aws-sdk/util-endpoints 3.511.0
    │       │   └── @smithy/util-endpoints 1.1.2
    │       ├─┬ @aws-sdk/util-endpoints 3.511.0
    │       │ └── @smithy/util-endpoints 1.1.2
    │       └── @smithy/util-endpoints 1.1.2
    └─┬ @aws-sdk/credential-provider-sso 3.511.0
      ├─┬ @aws-sdk/client-sso 3.511.0
      │ ├─┬ @aws-sdk/middleware-user-agent 3.511.0
      │ │ └─┬ @aws-sdk/util-endpoints 3.511.0
      │ │   └── @smithy/util-endpoints 1.1.2
      │ ├─┬ @aws-sdk/util-endpoints 3.511.0
      │ │ └── @smithy/util-endpoints 1.1.2
      │ └── @smithy/util-endpoints 1.1.2
      └─┬ @aws-sdk/token-providers 3.511.0
        └─┬ @aws-sdk/client-sso-oidc 3.511.0
          └─┬ @aws-sdk/middleware-user-agent 3.511.0
            └─┬ @aws-sdk/util-endpoints 3.511.0
              └── @smithy/util-endpoints 1.1.2
+2
seawatts on Feb 27, 2024

I have the same issue here… when I upgrade my project from Next.js 13 to Next.js 14. Unfortunately, I am trying to reproduce it on a small project, but there, it works like a charm so far 😭

Unlike the comments above, I only have a single version of @smithy/util-endpoints => 1.1.1.

However, I am using a monorepo with pnpm workspaces.

And the fun fact is that it works well with Next.js 13.5.6 but not when I upgrade only my next dependency to 14.1.0. I wonder if some bundler options somehow could also generate two occurrences of customEndpointFunctions??

Here is the call-stack if this can help…

    stack: 'TypeError: endpointFunctions[fn] is not a function\n' +
      '    at callFunction (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:258:33)\n' +
      '    at evaluateCondition (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:266:19)\n' +
      '    at evaluateConditions (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:283:38)\n' +
      '    at evaluateErrorRule (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:383:41)\n' +
      '    at evaluateRules (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:419:13)\n' +
      '    at evaluateTreeRule (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:402:12)\n' +
      '    at evaluateRules (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:421:41)\n' +
      '    at evaluateTreeRule (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:402:12)\n' +
      '    at evaluateRules (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:421:41)\n' +
      '    at resolveEndpoint (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+util-endpoints@1.1.1/node_modules/@smithy/util-endpoints/dist-cjs/index.js:452:22)\n' +
      '    at Object.defaultEndpointResolver [as endpointProvider] (webpack-internal:///(rsc)/../../node_modules/.pnpm/@aws-sdk+client-s3@3.499.0/node_modules/@aws-sdk/client-s3/dist-es/endpoint/endpointResolver.js:11:83)\n' +
      '    at getEndpointFromInstructions (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+middleware-endpoint@2.4.1/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:136:35)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@smithy+middleware-endpoint@2.4.1/node_modules/@smithy/middleware-endpoint/dist-cjs/index.js:172:30)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.499.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:107:28)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@aws-sdk+middleware-sdk-s3@3.499.0/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:132:24)\n' +
      '    at async eval (webpack-internal:///(rsc)/../../node_modules/.pnpm/@aws-sdk+middleware-logger@3.496.0/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:40:34)\n' +
      '    at async SkS3Bucket.exists (webpack-internal:///(rsc)/../../packages/aws/s3.ts:45:30)\n' +
+2
janvorwerk on Jan 26, 2024

Got same issue when using version 3.481.0, seems the recent 3.490.0 works. Both @aws-sdk/client-sns, @aws-sdk/client-sqs, @aws-sdk/client-s3 in same 490 version and that magically worked! 🤔

+2
josereymondaguilar on Jan 16, 2024

@sheeni17 What worked for me was to revert sst to version 2.35.1

+2
Makiavelo on Dec 11, 2023

@janvorwerk We are also facing the same issue. We tried pinning the version, but no success.

What worked for me @coronapl is to remain on an older AWS SDK version as mentionned in the bug description.

+1
janvorwerk on Mar 20, 2024

I have the same issue as https://github.com/aws/aws-sdk-js-v3/issues/5435#issuecomment-1854878590. It failed at handling fn = aws.partition.

Investigation

I can see the aws.partition is set up at https://github.com/aws/aws-sdk-js-v3/blob/main/packages/util-endpoints/src/aws.ts#L13

The logic is supposed to hit the branch https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-endpoints/src/utils/callFunction.ts#L13. But it hit https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-endpoints/src/utils/callFunction.ts#L16

From my local error stack: at callFunction (/<project-root-path>/node_modules/@aws-sdk/client-sts/node_modules/@smithy/util-endpoints/dist-cjs/index.js:271:31)

I can see there was a @smithy under @aws-sdk/client-sts/node_modules.

I believe this error is caused by the way we register customEndpointFunctions.aws. We update the customEndpointFunctions.aws from @smithy in aws-sdk.

I use sst to manage my stacks on top of CDK. They specify the @smithy/util-endpoints version as 1.0.5 in the latest sst version (2.39.6). https://github.com/sst/sst/commit/ff3abe36516b0e110978390178fe9c17fbac294e#diff-6a1935fc5ad82cc9ecc6277e01331eda95c2f1494e465569e5c325edcc8e596fR66 But the version in aws-sdk is 1.1.0.

Because of the version conflicts, @aws-sdk/client-sts and @aws-sdk/util-endpoints have their own @smithy/util-endpoints. Here is the layout for my local dependencies.

.
└── node_modules
    ├── @aws-sdk
    │   ├── client-sts
    │   │   └── node_modules
    │   │       └── @smithy
    │   │           └── util-endpoints (1.1.0)
    │   └── util-endpoints
    │       └── node_modules
    │           └── @smithy
    │               └── util-endpoints (1.1.0)
    └── @smithy
        └── util-endpoints (1.0.5, from sst)

In client-sts We import @aws-sdk/util-endpoints to register customerEndpointFunctions. (https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/src/index.ts#L19) The customerEndpointFunctions is stored in @aws-sdk/util-endpoints/node_modules/@smithy/util-endpoints.

But when we resolve the endpoint, we import the resolveEndpoint function from @smithy/util-endpoints directly. (https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-sts/src/endpoint/endpointResolver.ts#L12) Then, this resolveEndpoint function will access customerEndpointFunctions in @aws-sdk/client-sts/node_modules/@smithy/util-endpoints, which is different from the one we registered above. Then we will get an empty customerEndpointFunctions.

Possible Solution

So one possible solution is to re-export resolveEndpoint in @aws-sdk/util-endpoints since we register customEndpointFunctions in it. The call stack will look like: @aws-sdk/client-sts -> @aws-sdk/util-endpoints -> @smithy/util-endpoints. But, ideally, maybe we should find a better way to manage the customEndpointFunctions. 🤔

For a temp workaround, I revert sst to the previous version (2.39.5), delete the package-lock.json and node_modules, and re-run the build. It can work. The extra @smithy/util-endpoints in sub-modules are gone. That means we should make sure we are using the same version for the @smithy/util-endpoints in our project.

If I resume sst 2.39.6, delete the package-lock.json and node_modules, and re-build, I can replicate the error.

Hope this can help with your investigation. @RanVaknin

+1
changchang on Jan 19, 2024

I just ran into this issue when upgrading to sst@2.36.7 which upgraded to aws-cdk-lib@2.110.1.

I also have a module called aws (packages/core/aws to be specific) but renaming this to something else didn’t have any impact. @ytimenkov how exactly did you fix the issue for you scenario? did you just remove your custom AWS module entirely, or rename it and it still worked?

+1
kopertop on Nov 30, 2023

Had the same issue with @aws-sdk/client-sts reverting to 3.428.0 fixed the issue for now.

+1
nagarjuna993 on Nov 28, 2023

Hi @taoatmars,

I’m not able to reproduce the reported behavior. I used Docker to containerize my application and I can use the lambda client without any issues.

It’s possible that your lock file has multiple copies of the SDK pinned at different versions.

If you want to inspect it you can try this:

npm ls @aws-sdk/client-lambda

Or let npm resolve the dependency tree by:

cd your-project-root
rm -rf node_modules && rm package-lock.json && npm install

to regenerate your lock file.
Then rebuild your image and run it.

Let me know how it goes.

All the best, Ran~

+1
RanVaknin on Nov 2, 2023
Read more comments on GitHub
← aws-sdk-js-v3: EMFIL (too many open files) still exists when making mass S3 calls in JS V3 @aws-sdk/client-s3
aws-sdk-js-v3: Error Module has no exported member 'UserAgent' while using aws-sdk-js-v3 →