aws-sam-cli: Bug: SAM CLI from DevContainer fails either on Windows or Mac

Description:

When I want to run sam local start-api inside a DevContainer, I encounter different failures depending on what platform my host machine is in, and what configuration I have.

Steps to reproduce:

There are two variants (or scenarios) of configurations I can make. One that fails consistently on Windows, and one that fails consistently on Mac. But I can never have it not fail on both.

Scenario 1

Repro: https://github.com/ffMathy/aws-sam-cli-repro/tree/main (main branch)

• On Windows, the DevContainer itself crashes and disconnects (see attached screenshot below).
• On Mac, it works as expected.

Scenario 2

Repro: https://github.com/ffMathy/aws-sam-cli-repro/tree/windows_fix (windows_fix branch)

Diff from Scenario 1: https://github.com/ffMathy/aws-sam-cli-repro/compare/main...windows_fix

• On Windows, the SAM CLI gives a runtime error (error 500) and can’t find the application entrypoint, but at least it doesn’t crash the whole container (that’s what this issue is about).
• On Mac, I get Error: Lambda functions containers initialization failed.

Observed result:

Scenario 1 failure (Windows)

Scenario 2 failure (Mac)

2023-09-12 09:40:25,608 | Config file location: /workspace/samconfig.toml                                                                                                              
2023-09-12 09:40:25,672 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in config file at '/workspace/samconfig.toml'...    
2023-09-12 09:40:25,673 | Configuration values successfully loaded.                                                                                                                    
2023-09-12 09:40:25,673 | Configuration values are: {'stack_name': 'lambda-nodejs18.x', 'warm_containers': 'EAGER'}                                                                    
2023-09-12 09:40:25,843 | Using config file: samconfig.toml, config environment: default                                                                                               
2023-09-12 09:40:25,843 | Expand command line arguments to:                                                                                                                            
2023-09-12 09:40:25,844 | --template_file=/workspace/template.yaml --host=0.0.0.0 --container_host=host.docker.internal --container_host_interface=0.0.0.0                             
--docker_volume_basedir=/Users/dkMaLyLo/Documents/src/aws-sam-cli-repro --port=3000 --static_dir=public --layer_cache_basedir=/root/.aws-sam/layers-pkg --warm_containers=EAGER        
2023-09-12 09:40:25,946 | local start-api command is called                                                                                                                            
2023-09-12 09:40:25,965 | No Parameters detected in the template                                                                                                                       
2023-09-12 09:40:25,974 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id               
2023-09-12 09:40:25,974 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id                
2023-09-12 09:40:25,975 | 0 stacks found in the template                                                                                                                               
2023-09-12 09:40:25,975 | No Parameters detected in the template                                                                                                                       
2023-09-12 09:40:25,981 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id               
2023-09-12 09:40:25,982 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id                
2023-09-12 09:40:25,982 | 2 resources found in the stack                                                                                                                               
2023-09-12 09:40:25,982 | Found Serverless function with name='HelloWorldFunction' and CodeUri='/workspace/hello-world/'                                                               
2023-09-12 09:40:26,028 | watch resource /workspace/template.yaml                                                                                                                      
2023-09-12 09:40:26,029 | Create Observer for resource /workspace/template.yaml with recursive True                                                                                    
2023-09-12 09:40:26,030 | watch resource /workspace/template.yaml's parent /workspace                                                                                                  
2023-09-12 09:40:26,030 | Create Observer for resource /workspace with recursive False                                                                                                 
2023-09-12 09:40:26,092 | Initializing the lambda functions containers.                                                                                                                
2023-09-12 09:40:26,093 | Async execution started                                                                                                                                      
2023-09-12 09:40:26,093 | Invoking function functools.partial(<function InvokeContext._initialize_all_functions_containers.<locals>.initialize_function_container at 0xffffa8047ba0>,  
Function(function_id='HelloWorldFunction', name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='nodejs18.x', memory=None, timeout=3, handler='app.lambdaHandler',    
imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/workspace/hello-world/', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties':  
{'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata={'SamResourceId': 'HelloWorldFunction'}, inlinecode=None, codesign_config_arn=None,                  
architectures=['x86_64'], function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>,    
stack_path='', runtime_management_config=None))                                                                                                                                        
2023-09-12 09:40:26,096 | Waiting for async results                                                                                                                                    
2023-09-12 09:40:26,100 | No environment variables found for function 'HelloWorldFunction'                                                                                             
2023-09-12 09:40:26,101 | Loading AWS credentials from session with profile 'None'                                                                                                     
2023-09-12 09:40:26,276 | Resolving code path. Cwd=/Users/dkMaLyLo/Documents/src/aws-sam-cli-repro, CodeUri=/workspace/hello-world/                                                    
2023-09-12 09:40:26,277 | Resolved absolute path to code is /workspace/hello-world/                                                                                                    
2023-09-12 09:40:26,393 | watch resource /workspace/hello-world/                                                                                                                       
2023-09-12 09:40:26,393 | Create Observer for resource /workspace/hello-world/ with recursive True                                                                                     
2023-09-12 09:40:26,459 | watch resource /workspace/hello-world/'s parent /workspace                                                                                                   
2023-09-12 09:40:26,460 | Code /workspace/hello-world/ is not a zip/jar file                                                                                                           
2023-09-12 09:40:29,986 | Local image is out of date and will be updated to the latest runtime. To skip this, pass in the parameter --skip-pull-image                                  
Building image...........................................................................................................................................................................................................................................................................................................................................................................................................................
2023-09-12 09:41:10,808 | Using local image: public.ecr.aws/lambda/nodejs:18-rapid-x86_64.                                                                                             
                                                                                                                                                                                       
2023-09-12 09:41:10,810 | Mounting /workspace/hello-world/ as /var/task:ro,delegated, inside runtime container                                                                         
2023-09-12 09:41:11,120 | Exception raised during the execution                                                                                                                        
2023-09-12 09:41:11,121 | Lambda functions containers initialization failed because of 500 Server Error for                                                                            
http+docker://localhost/v1.35/containers/d55e542097167385c2c5d52cd3651592b12a3ccff5ac62ea4586d39f1cbce068/start: Internal Server Error ("error while creating mount source path        
'/host_mnt/workspace/hello-world': mkdir /host_mnt/workspace: read-only file system")                                                                                                  
2023-09-12 09:41:11,123 | Terminating all running warm containers                                                                                                                      
2023-09-12 09:41:11,123 | Terminate running warm container for Lambda Function 'HelloWorldFunction'                                                                                    
2023-09-12 09:41:11,273 | Cleaning all decompressed code dirs                                                                                                                          
2023-09-12 09:41:11,280 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                   
2023-09-12 09:41:11,857 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                   
2023-09-12 09:41:11,857 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '228a789b-748b-4bc2-84b8-c7eb8680e85b', 'installationId':                                       
'ac71eba1-d337-48e5-960e-714e9eda4f7e', 'sessionId': 'd9be9397-3c79-42a9-9394-490ec4bc799f', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.2', 'samcliVersion':       
'1.97.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local start-api', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin':  
None, 'projectName': '21a3230e03772a58aff1b3709a9e232850916337e1fba95c434076b6668c6e08', 'initialCommit': None}, 'duration': 45436, 'exitReason': 'ContainersInitializationException', 
'exitCode': 1}}]}                                                                                                                                                                      
2023-09-12 09:41:11,858 | Unable to find Click Context for getting session_id.                                                                                                         
2023-09-12 09:41:11,860 | Sending Telemetry: {'metrics': [{'events': {'requestId': '5d1e658a-a542-4ab2-9d04-71573b0bcec7', 'installationId': 'ac71eba1-d337-48e5-960e-714e9eda4f7e',   
'sessionId': 'd9be9397-3c79-42a9-9394-490ec4bc799f', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.2', 'samcliVersion': '1.97.0', 'commandName': 'sam local           
start-api', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '045bb4ff0e3646108ff1d543c4af58c8', 'time_stamp':    
'2023-09-12 09:40:25.607', 'exception_name': None}]}}}]}                                                                                                                               
2023-09-12 09:41:12,616 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                             
2023-09-12 09:41:12,617 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)                             
Error: Lambda functions containers initialization failed

Expected result:

I expected the SAM CLI to be launchable from inside Docker. Docker is heavily used for most developers, especially dev-containers.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows or Mac
2. sam --version: SAM CLI, version 1.97.0
3. AWS region: eu-west-1

{
  "version": "1.97.0",
  "system": {
    "python": "3.11.2",
    "os": "Linux-5.15.49-linuxkit-pr-aarch64-with-glibc2.36"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.5",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}