pywren: [Solution Included]pywren-config default is not properly setuped: making NoSuchBucket error.

Environ

• OS X 10.12.6 ( Sierra )
• Python3 (aws-cli installed/setuped)

Situation

When I followed getting started, I setup aws correctly however it doesn’t work, providing “NoSuchBucket” Error.

Error Trace log

$ pywren-setup
This is the PyWren interactive setup script
Your AWS configuration appears to be set up, and your account ID is xxxxxxxxxxxx
This interactive script will set up your initial PyWren configuration.
If this is the first time you are using PyWren then accepting the defaults should be fine.
What is your default aws region? [us-west-2]: ap-northeast-2
Location for config file:  [/Users/beomi/.pywren_config]:
PyWren requires an s3 bucket to store intermediate data. What s3 bucket would you like to use? [beomi-pywren-xxx]:
Bucket does not currently exist, would you like to create it? [Y/n]: y
PyWren prefixes every object it puts in S3 with a particular prefix.
PyWren s3 prefix:  [pywren.jobs]: lamlam
Would you like to configure advanced PyWren properties? [y/N]: n
PyWren standalone mode uses dedicated AWS instances to run PyWren tasks. This is more flexible, but more expensive with fewer simultaneous workers.
Would you like to enable PyWren standalone mode? [y/N]: n
Creating config /Users/beomi/.pywren_config
new default file created in /Users/beomi/.pywren_config
lambda role is pywren_exec_role_1
Creating bucket beomi-pywren-xxx.
Creating role.
Deploying lambda.
Pausing for 5 seconds for changes to propagate.
Pausing for 5 seconds for changes to propagate.
Pausing for 5 seconds for changes to propagate.
Successfully created function.
Pausing for 10 sec for changes to propagate.
Traceback (most recent call last):
  File "/Users/beomi/.virtualenvs/tmp/bin/pywren-setup", line 11, in <module>
    sys.exit(interactive_setup())
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/scripts/setupscript.py", line 208, in interactive_setup
    ctx.invoke(pywrencli.test_function)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/scripts/pywrencli.py", line 374, in test_function
    wrenexec = pywren.default_executor(config=config)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/wren.py", line 21, in default_executor
    return lambda_executor(**kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/wren.py", line 37, in lambda_executor
    return Executor(invoker, config, job_max_runtime)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/executor.py", line 40, in __init__
    self.runtime_meta_info = runtime.get_runtime_info(config['runtime'])
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/runtime.py", line 10, in get_runtime_info
    runtime_meta = storage.get_runtime_info(runtime_config)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/storage/storage.py", line 112, in get_runtime_info
    json_str = handler.get_object(key)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/storage/s3_backend.py", line 42, in get_object
    raise e
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/pywren/storage/s3_backend.py", line 35, in get_object
    r = self.s3client.get_object(Bucket=self.s3_bucket, Key=key)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/botocore/client.py", line 312, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/beomi/.virtualenvs/tmp/lib/python3.5/site-packages/botocore/client.py", line 605, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist

Solution

As Trace log says “NoSuchBucket”, So I changed my .pywren_config file’s settings.

Note: xxxxx masking for security.

Before

account:
    aws_account_id: xxxxxxxxxx
    aws_lambda_role: pywren_exec_role_1
    aws_region: ap-northeast-2


lambda:
    memory : 1536
    timeout : 300
    function_name : pywren_1

s3:
    bucket: beomi-pywren-xxx
    pywren_prefix: lamlam

runtime:
    s3_bucket: pywren-public-ap-northeast-2
    s3_key: pywren.runtimes/default_3.5.meta.json

scheduler:
    map_item_limit: 10000

standalone:
    ec2_instance_type: m4.4xlarge
    sqs_queue_name: pywren-jobs-1
    visibility: 10
    ec2_ssh_key : xxxxxxxxx
    target_ami : ami-xxxxxxxxx
    instance_name: pywren-standalone
    instance_profile_name: pywren-standalone
    max_idle_time: 60
    idle_terminate_granularity: 3600

After

account:
    aws_account_id: xxxxxxxxxx
    aws_lambda_role: pywren_exec_role_1
    aws_region: ap-northeast-2


lambda:
    memory : 1536
    timeout : 300
    function_name : pywren_1

s3:
    bucket: beomi-pywren-xxx
    pywren_prefix: lamlam

runtime:
    s3_bucket: pywren-public-us-west-2
    s3_key: pywren.runtimes/default_3.5.meta.json

scheduler:
    map_item_limit: 10000

standalone:
    ec2_instance_type: m4.4xlarge
    sqs_queue_name: pywren-jobs-1
    visibility: 10
    ec2_ssh_key : xxxxxxxxx
    target_ami : ami-xxxxxxxxx
    instance_name: pywren-standalone
    instance_profile_name: pywren-standalone
    max_idle_time: 60
    idle_terminate_granularity: 3600

As you see, I’ve changed runtime settings from runtime: s3_bucket: pywren-public-ap-northeast-2 to runtime: s3_bucket: pywren-public-us-west-2, then It worked properly.