prefect: OS Permission Error when running non-root prefect server due to changes from #11493

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn’t find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

#11493 introduces changes that cannot run in non-root docker environments. The problem is that the directory where the UI files are copied into, is not writeable in scenarios where the Docker user is non-root. This is the case when you override the User at the end of a Dockerfile. Deploying via helm using prefect-helm is also resulting in the same error due to the securityContext being set to User 1001

Reproduction

starting the docker image without privileges and command `prefect server start --port 4200`

Error

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/__main__.py", line 4, in <module>
    uvicorn.main()
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
    run(
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
    server.run()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/config.py", line 473, in load
    self.loaded_app = self.loaded_app()
  File "/opt/venv/lib/python3.10/site-packages/prefect/server/api/server.py", line 622, in create_app
    ui_app = create_ui_app(ephemeral)
  File "/opt/venv/lib/python3.10/site-packages/prefect/server/api/server.py", line 392, in create_ui_app
    create_ui_static_subpath()
  File "/opt/venv/lib/python3.10/site-packages/prefect/server/api/server.py", line 368, in create_ui_static_subpath
    os.makedirs(prefect.__ui_static_subpath__)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/opt/venv/lib/python3.10/site-packages/prefect/server/ui_build'
Server stopped!

Error mounting emptyDir with correct permissions

This error occurs if a emptyDir is mounted into the Pod which has the correct user permissions:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/__main__.py", line 4, in <module>
    uvicorn.main()
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main
    run(
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run
    server.run()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve
    config.load()
  File "/opt/venv/lib/python3.10/site-packages/uvicorn/config.py", line 473, in load
    self.loaded_app = self.loaded_app()
  File "/opt/venv/lib/python3.10/site-packages/prefect/server/api/server.py", line 622, in create_app
    ui_app = create_ui_app(ephemeral)
  File "/opt/venv/lib/python3.10/site-packages/prefect/server/api/server.py", line 392, in create_ui_app
    create_ui_static_subpath()
  File "/opt/venv/lib/python3.10/site-packages/prefect/server/api/server.py", line 367, in create_ui_static_subpath
    shutil.rmtree(prefect.__ui_static_subpath__)
  File "/usr/local/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/local/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 30] Read-only file system: '/opt/venv/lib/python3.10/site-packages/prefect/server/ui_build'
Server stopped!

Versions

python 3.10
prefect==2.4.15

Additional context

The error is fixable by being able to override the variable prefect.__ui_static_subpath__ either via prefect.settings or environment variable. That way I am able to setup an emptyDir with proper permissions. for that to work the emptyDir should not be removed or a subpath should be created in that directory for which you copy the contents of prefect/server/ui/ into.

Last known stable prefect version: 2.14.13