prefect: Flow could not be retrieved from deployment with s3fs==2023.3.0

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn’t find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

I started to get a “Flow could not be retrieved from deployment” error for a flow run deployment which had been previously working.

After several hours of investigation, I tracked it down a trailing slash required for S3.bucket_path with dependency s3fs==2023.3.0. A trailing slash was not required under s3fs==2023.1.0.

By simply adding a trailing slash to the S3.bucket_path, everything seems to work again.

Reproduction

from prefect import flow, get_run_logger
from prefect.deployments import Deployment
from prefect.filesystems import S3
from prefect.infrastructure import Process
from pathlib import Path

@flow
def my_flow():
    logger = get_run_logger()
    logger.info("Hello")
    return True

# The deployment flow run fails without a trailing slash in the `bucket_path` on s3fs==2023.3.0
# The deployment flow run succeeds without a trailing slash in the `bucket_path` on s3fs==2023.1.0
storage = S3(bucket_path = 'my-bucket/test-flow')

working_dir = Path('./working_dir').resolve()
working_dir.mkdir(exist_ok = True)
infrastructure = Process(working_dir=working_dir)

my_deployment = Deployment.build_from_flow(
    flow = my_flow,
    name = 'my-deployment',
    output = 'deployment.yaml',
    storage = storage,
    infrastructure = infrastructure,
    skip_upload = True,
    apply = False
)

if __name__ == '__main__':
    my_deployment.apply(upload = True)

Error

Downloading flow code from storage at None

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 879, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1016, in get_code
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/Users/lmendelowitz/playground/prefect/storage_issue/working_dir/deployment.py'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/engine.py", line 270, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/deployments.py", line 187, in load_flow_from_flow_run
    flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 91, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/flows.py", line 809, in load_flow_from_entrypoint
    flow = import_object(entrypoint)
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 197, in import_object
    module = load_script_as_module(script_path)
  File "/Users/lmendelowitz/.local/share/virtualenvs/storage_issue-SDGII8WY/lib/python3.10/site-packages/prefect/utilities/importtools.py", line 160, in load_script_as_module
    raise ScriptError(user_exc=exc, path=path) from exc
prefect.exceptions.ScriptError: Script at 'deployment.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')

Versions

Version:             2.8.3
API version:         0.8.4
Python version:      3.10.8
Git commit:          07de1ac4
Built:               Thu, Feb 23, 2023 4:31 PM
OS/Arch:             darwin/arm64
Profile:             local
Server type:         server

Additional context

Here are the requirements.txt:

-i https://pypi.org/simple
aiobotocore==2.4.2 ; python_version >= '3.7'
aiohttp==3.8.4 ; python_version >= '3.6'
aioitertools==0.11.0 ; python_version >= '3.6'
aiosignal==1.3.1 ; python_version >= '3.7'
aiosqlite==0.18.0 ; python_version >= '3.7'
alembic==1.10.0 ; python_version >= '3.7'
anyio==3.6.2 ; python_full_version >= '3.6.2'
appnope==0.1.3 ; sys_platform == 'darwin'
apprise==1.3.0 ; python_version >= '3.6'
asgi-lifespan==2.0.0 ; python_version >= '3.7'
asttokens==2.2.1
async-timeout==4.0.2 ; python_version >= '3.6'
asyncpg==0.27.0 ; python_full_version >= '3.7.0'
attrs==22.2.0 ; python_version >= '3.6'
backcall==0.2.0
botocore==1.27.59 ; python_version >= '3.7'
cachetools==5.3.0 ; python_version ~= '3.7'
certifi==2022.12.7 ; python_version >= '3.6'
cffi==1.15.1
charset-normalizer==3.0.1
click==8.1.3 ; python_version >= '3.7'
cloudpickle==2.2.1 ; python_version >= '3.6'
colorama==0.4.6 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6'
coolname==2.2.0
croniter==1.3.8 ; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
cryptography==39.0.2 ; python_version >= '3.6'
dateparser==1.1.7 ; python_version >= '3.7'
decorator==5.1.1 ; python_version >= '3.5'
docker==6.0.1 ; python_version >= '3.7'
executing==1.2.0
fastapi==0.92.0 ; python_version >= '3.7'
frozenlist==1.3.3 ; python_version >= '3.7'
fsspec==2023.3.0 ; python_version >= '3.8'
google-auth==2.16.2 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
greenlet==2.0.2
griffe==0.25.5 ; python_version >= '3.7'
h11==0.14.0 ; python_version >= '3.7'
h2==4.1.0
hpack==4.0.0 ; python_full_version >= '3.6.1'
httpcore==0.16.3 ; python_version >= '3.7'
httpx[http2]==0.23.3 ; python_version >= '3.7'
hyperframe==6.0.1 ; python_full_version >= '3.6.1'
idna==3.4 ; python_version >= '3.5'
ipython==8.11.0
jedi==0.18.2 ; python_version >= '3.6'
jinja2==3.1.2 ; python_version >= '3.7'
jmespath==1.0.1 ; python_version >= '3.7'
jsonpatch==1.32 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
jsonpointer==2.3 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
jsonschema==4.17.3 ; python_version >= '3.7'
kubernetes==26.1.0 ; python_version >= '3.6'
mako==1.2.4 ; python_version >= '3.7'
markdown==3.4.1 ; python_version >= '3.7'
markdown-it-py==2.2.0 ; python_version >= '3.7'
markupsafe==2.1.2 ; python_version >= '3.7'
matplotlib-inline==0.1.6 ; python_version >= '3.5'
mdurl==0.1.2 ; python_version >= '3.7'
multidict==6.0.4 ; python_version >= '3.7'
oauthlib==3.2.2 ; python_version >= '3.6'
orjson==3.8.7 ; python_version >= '3.7'
packaging==23.0 ; python_version >= '3.7'
parso==0.8.3 ; python_version >= '3.6'
pathspec==0.11.0 ; python_version >= '3.7'
pendulum==2.1.2 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
pexpect==4.8.0 ; sys_platform != 'win32'
pickleshare==0.7.5
prefect==2.8.3
prompt-toolkit==3.0.38 ; python_full_version >= '3.7.0'
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydantic==1.10.5 ; python_version >= '3.7'
pygments==2.14.0 ; python_version >= '3.6'
pyrsistent==0.19.3 ; python_version >= '3.7'
python-dateutil==2.8.2 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
python-slugify==8.0.1 ; python_version >= '3.7'
pytz==2022.7.1
pytz-deprecation-shim==0.1.0.post0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
pytzdata==2020.1 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
pyyaml==6.0 ; python_version >= '3.6'
readchar==4.0.3 ; python_version >= '3.6'
regex==2022.10.31 ; python_version >= '3.6'
requests==2.28.2 ; python_version >= '3.7' and python_version < '4'
requests-oauthlib==1.3.1 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
rfc3986[idna2008]==1.5.0
rich==13.3.2 ; python_full_version >= '3.7.0'
rsa==4.9 ; python_version >= '3.6'
s3fs==2023.3.0
setuptools==67.5.0 ; python_version >= '3.7'
six==1.16.0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
sniffio==1.3.0 ; python_version >= '3.7'
sqlalchemy[asyncio]==1.4.46 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
stack-data==0.6.2
starlette==0.25.0 ; python_version >= '3.7'
text-unidecode==1.3
toml==0.10.2 ; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
traitlets==5.9.0 ; python_version >= '3.7'
typer==0.7.0 ; python_version >= '3.6'
typing-extensions==4.5.0 ; python_version >= '3.7'
tzdata==2022.7 ; python_version >= '3.6'
tzlocal==4.2 ; python_version >= '3.6'
urllib3==1.26.14 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
uvicorn==0.20.0 ; python_version >= '3.7'
wcwidth==0.2.6
websocket-client==1.5.1 ; python_version >= '3.7'
websockets==10.4 ; python_version >= '3.7'
wrapt==1.15.0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
yarl==1.8.2 ; python_version >= '3.7'