aws-sam-cli: sam build does not recognise local nodejs projects outside CodeUri

Description

I have 2 Node.js projects: CommonT and Profile. Profile has a local dependency on CommonT.

When running the sam build -t template.yaml --debug command, it fails with the following error:

npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

Here is the project structure:

├── Backend
│   ├── CommonT
│   │   ├── package.json
│   │   └── src
│   └── Profile
│       ├── package.json
│       └── src
└── template.yaml

Here is the Profile/package.json:

{
  "main": "dist/src/index.js",
  ...
  "scripts": {
    "test": "ts-node ./node_modules/jasmine/bin/jasmine",
    "build": "tsc",
    "deploy": "npm run build && sls deploy"
  },
  "dependencies": {
    "@myteam/common": "file:../CommonT",
	...
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.46",
    "@types/node": "^12.12.32",
    "aws-sdk": "^2.508.0",
    "ts-node": "^8.10.2",
    "typescript": "^3.8.3",
	...
  }
}

Here is the CommonT/package.json:

{
  "name": "@myteam/common",
  ...
  "main": "dist/src/index",
  "typings": "dist/src/index",
  "private": true,
  "scripts": {
    "test": "ts-node ./node_modules/jasmine/bin/jasmine",
    "build": "tsc",
    "deploy": "npm run build && sls deploy"
  },
  "dependencies": {
    ...
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.46",
    "@types/node": "^12.12.32",
    "aws-sdk": "^2.508.0",
    "ts-node": "^8.10.2",
    "typescript": "^3.8.3",
    ...
  }
}

Here is my template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  My APIs
 
Resources:
  Profile:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/Profile
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetOneUser:
          Type: Api
          Properties:
            ...
  ...

This is an error I was facing with sam --version: 0.11.0 that disapeared when upgrading to sam --version: 0.52.0 using homebrew. I am currently moving CICD tool and I installed the same sam --version: 0.52.0 using pip3 as I cannot install homebrew on my CICD platform. So here it is:

npm build run manually: works fine
sam build (v 0.11.0) - Homebrew install: ERR! code ENOLOCAL
sam build (v 0.52.0) - Homebrew install: works fine
sam build (v 0.52.0) - pip3 install: ERR! code ENOLOCAL

Steps to reproduce

cd __project_root__

npm install --prefix Backend/CommonT
npm run build --prefix Backend/CommonT
npm run test --prefix Backend/CommonT

npm install --prefix Backend/Profile
npm run build --prefix Backend/Profile
npm run test --prefix Backend/Profile

# I tried both with and without this line, it doesn't make a difference
# npm install -q --no-audit --no-save --production --unsafe-perm --prefix %ROOT%/CommonT

# This is the line that fails
sam build -t template.yaml --debug

Observed result

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {}
8 resources found in the template
Found Serverless function with name='Profile' and CodeUri='Backend/Profile'
Found Serverless function with name='###Something else###' and CodeUri='###'
Collected default values for parameters: {}
Building function 'Profile'
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'NodejsNpmBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Running workflow 'NodejsNpmBuilder'
Running NodejsNpmBuilder:NpmPack
NODEJS packaging file:/my/directory/job/Backend/Profile to /my/directory/temp/buildTmp/XXXXX
executing NPM: ['npm', 'pack', '-q', 'file:/my/directory/job/Backend/Profile']
NODEJS packed to canaree-profile-1.0.0.tgz
NODEJS extracting to /my/directory/temp/buildTmp/XXXXX/unpacked
NodejsNpmBuilder:NpmPack succeeded
Running NodejsNpmBuilder:CopyNpmrc
.npmrc copying in: /my/directory/temp/buildTmp/XXXXX/unpacked/package
NodejsNpmBuilder:CopyNpmrc succeeded
Running NodejsNpmBuilder:CopySource
NodejsNpmBuilder:CopySource succeeded
Running NodejsNpmBuilder:NpmInstall
NODEJS installing in: /my/directory/job/.aws-sam/build/Profile
executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--production', '--unsafe-perm']
NodejsNpmBuilder:NpmInstall failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 108, in execute
    ["install", "-q", "--no-audit", "--no-save", "--production", "--unsafe-perm"], cwd=self.artifacts_dir
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/npm.py", line 85, in run
    raise NpmExecutionError(message=err.decode("utf8").strip())
aws_lambda_builders.workflows.nodejs_npm.npm.NpmExecutionError: NPM Failed: npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-06-22T12_03_11_067Z-debug.log

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflow.py", line 269, in run
    action.execute()
  File "/usr/local/lib/python3.6/dist-packages/aws_lambda_builders/workflows/nodejs_npm/actions.py", line 112, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: NPM Failed: npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log

Build Failed

npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 2034, 'exitReason': 'WorkflowFailedError', 'exitCode': 1, 'requestId': '42b13729-2136-4df8-8442-a25c0ae2ce19', 'installationId': 'acac1f1a-some-long-id-b854a9c82c0b', 'sessionId': 'fb6bba5f-some-other-long-id-ed535022cafb', 'executionEnvironment': 'CLI', 'pyversion': '3.6.9', 'samcliVersion': '0.52.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: NodejsNpmBuilder:NpmInstall - NPM Failed: npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm ERR! code ENOLOCAL
npm ERR! Could not install from "../CommonT" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/XXXXXXXX-debug.log

Expected result

When manually running the builds/tests with npm, everything compiles and run properly. I expect the same behavour to happen with sam build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Ubuntu 18.04.4 LTS
sam --version: 0.52.0

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 18
Comments: 18 (5 by maintainers)

Most upvoted comments

Here is a work around:

I did manage to “force” the creation of the CommonT package by adding a resource with the CommonT name in my template.yaml.

Here is my new file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  My APIs
 
Resources:
  CommonT: # <= This NEEDS to have the same name as the CommonT Typescript project!
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/CommonT
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetCommonData:
          Type: Api
          Properties:
            RestApiId: !Ref RestAPI
            Path: /some/config/data/for/example
            Method: get

  Profile:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: Backend/Profile
      Handler: dist/src/index.handler
      Runtime: nodejs12.x
      Events:
        GetOneUser:
          Type: Api
          Properties:
            ...
  ...

Then, when sam build creates the .aws-sam package, it starts by creating the folder CommonT for the resource with the same name. And it is then used as the common project as well.

It is dirty… but it allows to deploy for now.

rfrad on Jul 10, 2020

hi, are there any news reagrding that topic? we are highly interested in that beeing solved.

kellerkind87 on Aug 12, 2021

@cBiscuitSurprise that idea is quite the last option we should consider in our case. local modules are standard in npm so it should just work. the impact in introducing new registries for npm packages is quite big - credentials, build pipelines, local build etc. But at least it is a valid option.

what we have done so far, is to create a symlink for our local packages into the aws build dir. at least it is working, but still not a very good option, because there is no optimization within the package steps. and you have to put this peace of linking code to each function.

{
...

"scripts": {
    "preinstall": "node -e \"const fs = require('fs'); const path = require('path'); if (process.env.sam_build && !fs.existsSync('../../lib')) { fs.symlinkSync((process.env.CODEBUILD_SRC_DIR || '..') +'/lib/', path.resolve('../../lib/'), 'junction')}\""
  },
...
}

with that kind of workaround we can build and package using sam build on local machine and with cloudbuild.

In the end we need the support for local modules. 😃

kellerkind87 on Aug 25, 2021

Thanks, @bingtimren & @cBiscuitSurprise, for the replies! Sadly we are not using CDK for various reasons.

my team has been slowly making proper JS packages out of all the instances where we’re using this feature.

With that, do you mean a package that is published to the npm repository? Unfortunately, our internal library is private and needs to be resolved by our build system and other dev machines. Is there a way to get around this?

My goal is to share a library between all of my lambda functions. I honestly don’t have a preference on how it’s happening.

eladg on Aug 17, 2021

Here’s my work around. Simply put - copy the dependency project into the current root.

My project has three parts - frontend, backend (SAM) and “proj-common”, common artifacts for both front and backend.

I first put the dependency in package.json “dependencies” as “file:…/proj-commons”, pointing to the local folder. This works for the IDE, but sam build fails. The reason is SAM build in a container. In that container the mapped dir cannot access proj-commons. Check .aws-sam/build dir for the files actually available in SAM build.

I tried to put “…/proj-commons” in “files” of “package.json” and it does not work. I tried to symbol link “…/proj-commons” to a folder under the root, and refer to the folder in both “dependencies” and “files”, it does not work.

So the final work around is - in my build script, first copy “…/proj-commons” into a directory under the SAM project root, and also put it inside “files” in “project.json”.

Here’s how my package.json looks like. Note the build script.

    "dependencies": {
        "aws-sdk": "^2.437.0",
        "proj-commons": "file:./.dependency_by_copy/proj-commons"
    },
    "files": [
        "built/",
        ".dependency_by_copy/"
    ],
....
    "scripts": {
        "build": ". ./env.sh; rm -fr ./.dependency_by_copy; mkdir .dependency_by_copy; cp -r ../proj-commons ./.dependency_by_copy; tsc; sam build; announce \"Project Built\"",

bingtimren on Oct 6, 2020

@eladg Yep, we’re slowly isolating our modules as needed into packages which we publish to our private npm package registry (we use GitLab’s package registry – it can be free, even for private registries) and then just installing them like anything else.

cBiscuitSurprise on Aug 18, 2021

If you are willing to try CDK, NodejsFunction with esbuild works for me pretty well.

See

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#local-bundling

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-nodejs-readme.html#configuring-esbuild

bingtimren on Aug 14, 2021

I am trying to add some shared code to a layer but I am getting this error as well.

The yarn link did not worked for me. But @bingtimren workaround did, thank you !

Before :

{
  ...
  "scripts": {
    "build": "yarn install"
  },
  "dependencies": {
    "shared": "file:./../shared"
  }
}

After :

{
  ...
  "scripts": {
    "build": "mkdir -p dist && rsync -avr ../../shared dist --exclude node_modules --exclude src && yarn install"
  },
  "dependencies": {
    "shared": "file:./dist/shared"
  },
  "files": [
    "dist"
  ]
}

You can find the full example here : https://github.com/fbn-roussel/aws-sam-typescript-webpack-backend/blob/main/layers/application-layer/package.json

Native support of local path dependency would be greatly appreciated 😃

fbn-roussel on Dec 17, 2020