prefect: Inconsistent configuration of S3Result vs S3 Storage with custom S3 backend
Description
The following S3 storage configuration works as expected when connecting to Minio:
storage = S3(
bucket="flows",
aws_access_key_id=os.getenv("MY_MINIO_ID"),
aws_secret_access_key=os.getenv("MY_MINIO_KEY"),
client_options=dict(endpoint_url=os.getenv("MINIO_ENDPOINT")),
)
Although when trying to work with S3Result
:
result = S3Result(
bucket="results",
boto3_kwargs=dict(
aws_access_key_id=os.getenv("MY_MINIO_ID"),
aws_secret_access_key=os.getenv("MY_MINIO_KEY"),
client_options=dict(endpoint_url=os.getenv("MINIO_ENDPOINT")),
),
)
I receive the following error in UI:
prefect-agent_1 | [2020-06-09 07:59:36] DEBUG - prefect.S3Result | Starting to upload result to 2020/6/9/5a96e8ef-70d1-4d47-a495-15b0afa99169.prefect_result...
prefect-agent_1 | [2020-06-09 07:59:36] ERROR - prefect.CloudTaskRunner | Unexpected error: TypeError("client() got multiple values for keyword argument 'aws_access_key_id'")
prefect-agent_1 | Traceback (most recent call last):
prefect-agent_1 | File "/usr/local/lib/python3.7/site-packages/prefect/engine/runner.py", line 48, in inner
prefect-agent_1 | new_state = method(self, state, *args, **kwargs)
prefect-agent_1 | File "/usr/local/lib/python3.7/site-packages/prefect/engine/task_runner.py", line 986, in get_task_run_state
prefect-agent_1 | result = self.result.write(value, filename="output", **prefect.context)
prefect-agent_1 | File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/s3_result.py", line 103, in write
prefect-agent_1 | self.client.upload_fileobj(stream, Bucket=self.bucket, Key=new.location)
prefect-agent_1 | File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/s3_result.py", line 60, in client
prefect-agent_1 | self.initialize_client()
prefect-agent_1 | File "/usr/local/lib/python3.7/site-packages/prefect/engine/results/s3_result.py", line 49, in initialize_client
prefect-agent_1 | "s3", credentials=None, use_session=True, **self.boto3_kwargs
prefect-agent_1 | File "/usr/local/lib/python3.7/site-packages/prefect/utilities/aws.py", line 49, in get_boto_client
prefect-agent_1 | **kwargs
prefect-agent_1 | TypeError: client() got multiple values for keyword argument 'aws_access_key_id'
Expected Behavior
Expectation is to be able to configure S3Result by directly passing boto3 arguments (same as it works for S3 storage) and have consistency across these 2 structures.
Reproduction
Sample flow I’m using:
import os
from prefect import task, Flow
from prefect.engine.results.s3_result import S3Result
from prefect.environments.storage import Docker
@task
def add(x, y=1):
"""
The only task we use so far here ;-)
"""
return x + y
def create_flow():
"""
Create the flow
"""
result = S3Result(
bucket="results",
boto3_kwargs=dict(
aws_access_key_id=os.getenv("MY_MINIO_ID"),
aws_secret_access_key=os.getenv("MY_MINIO_KEY"),
client_options=dict(endpoint_url=os.getenv("MINIO_ENDPOINT")),
),
)
with Flow("Sample Flow", result=result) as flow:
first_result = add(1, y=2)
second_result = add(x=first_result, y=100)
storage = Docker()
storage.add_flow(flow)
flow.storage = storage
return flow
Environment
Any additional information about your environment
- OSX
- Docker Compose
- Docker Agent
Optionally run prefect diagnostics
from the command line and paste the information here
root@9675cb8de5d4:/opt/packages# prefect diagnostics
{
"config_overrides": {},
"env_vars": [
"PREFECT__LOGGING__LEVEL",
"PREFECT__SERVER__HOST",
"PREFECT__BACKEND"
],
"system_information": {
"platform": "Linux-4.19.76-linuxkit-x86_64-with-debian-10.4",
"prefect_version": "0.11.5",
"python_version": "3.7.7"
}
}
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18
@oleksandr Yeah that could be it. Removing that and also adding
-U
to the pip install should install it from the branch.