aws-sam-cli: Bug: Sam Build very slow and ignoring cache, recopying unchanged files and dependencies

Description:

For context project is approx 40 lambdas, all python 3.8, using dependency layer for requirements.

Running sam build has become incredibly slow (10-15 min on my Macbook Pro) as project has grown and is really slowing down development process. Running sam build --parallel --debug --cached shows that most of the time seems to be taken during the Copying source file steps.

My lambda code is only about 1.5MB, the dependency layer is near 256MB. It appears that the Copy Source step is recopying the .aws-sam/deps/{path to dependency cache file} to the .aws-sam/build/DependencyLayer/{path} even when the dependency layer is unchanged.

This happens even if i just make a change to the template and no code changes, i still have to wait for this entire 10+ min build.

Steps to reproduce:

Here is the relevant bit of my template for the dependency layer

  PipDependencyLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: dependencies
      CompatibleRuntimes:
        - python3.8
    Metadata:
      BuildMethod: python3.8

I have a dependencies folder in my working directory root (same level as my project root)

dependencies
  - python
  -  requirements.txt
project_root
  - lambda_handler_1.py
  - ... other project files

Requirements.txt content is below

-i https://pypi.org/simple
arabic-reshaper==2.1.4
asn1crypto==1.5.1
attrs==22.1.0; python_version >= '3.5'
aws-encryption-sdk==3.1.1
backports.zoneinfo==0.2.1; python_version < '3.9'
boto3==1.26.18; python_version >= '3.7'
botocore==1.29.18; python_version >= '3.7'
certifi==2022.9.24; python_version >= '3.6'
cffi==1.15.1
charset-normalizer==2.1.1; python_version >= '3.6'
click==8.1.3; python_version >= '3.7'
cryptography==38.0.4; python_version >= '3.6'
cssselect2==0.7.0; python_version >= '3.7'
future==0.18.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
html5lib==1.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
idna==3.4; python_version >= '3.5'
jmespath==1.0.1; python_version >= '3.7'
lxml==4.9.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
numpy==1.23.5
oscrypto==1.3.0
pandas==1.5.2
pillow==9.3.0
pycparser==2.21
pyhanko==0.15.1
pyhanko-certvalidator==0.19.6
pyjwt==2.6.0
pypdf3==1.0.6
python-bidi==0.4.2
python-dateutil==2.8.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
python-http-client==3.3.7; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
python-liquid==1.4.7
pytz==2022.6
pytz-deprecation-shim==0.1.0.post0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
pyyaml==6.0; python_version >= '3.6'
qrcode==7.3.1; python_version >= '3.6'
ratelimit==2.2.1
reportlab==3.6.12; python_version >= '3.7' and python_version < '4'
requests==2.28.1
s3transfer==0.6.0; python_version >= '3.7'
sendgrid==6.9.7
simplejson==3.18.0
six==1.16.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
starkbank-ecdsa==2.2.0
stripe==5.0.0
svglib==1.4.1; python_version >= '3.7'
tinycss2==1.2.1; python_version >= '3.7'
tqdm==4.64.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
typing-extensions==4.4.0; python_version >= '3.7'
tzdata==2022.6; python_version >= '3.6'
tzlocal==4.2; python_version >= '3.6'
uritools==4.0.0; python_version ~= '3.7'
urllib3==1.26.13; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
webencodings==0.5.1
wrapt==1.14.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
xhtml2pdf==0.2.8

Observed result:

Described in the description, the build is very slow

Expected result:

• I would expect that unchanged dependency layer would not be recopied into the .aws-sam/build directory even if lambda changes are made as long as no dependencies are updated
• I would expect that if no code changes are made and only template updates completed that there would be no build/copy source step necessary

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

output sam --info

{
  "version": "1.76.0",
  "system": {
    "python": "3.8.16",
    "os": "macOS-12.5.1-x86_64-i386-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  }
}