prefect: Broken docker-compose.yml
Description
prefect server start fails since 0.14.7 release.
Expected Behavior
Prefect UI starting.
Observed behaviour
Execution fails with
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.ui.depends_on contains an invalid type, it should be an array
services.apollo.depends_on contains an invalid type, it should be an array
services.towel.depends_on contains an invalid type, it should be an array
services.graphql.depends_on contains an invalid type, it should be an array
services.hasura.depends_on contains an invalid type, it should be an array
Exception caught; killing services (press ctrl-C to force)
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.ui.depends_on contains an invalid type, it should be an array
services.apollo.depends_on contains an invalid type, it should be an array
services.towel.depends_on contains an invalid type, it should be an array
services.graphql.depends_on contains an invalid type, it should be an array
services.hasura.depends_on contains an invalid type, it should be an array
Traceback (most recent call last):
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/prefect/cli/server.py", line 346, in start
subprocess.check_call(
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vincentchery/miniconda3/envs/prefect_test/bin/prefect", line 10, in <module>
sys.exit(cli())
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/site-packages/prefect/cli/server.py", line 384, in start
subprocess.check_output(
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/subprocess.py", line 411, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/home/vincentchery/miniconda3/envs/prefect_test/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker-compose', 'down']' returned non-zero exit status 1.
Reproduction
- Install prefect >= 0.14.7
- run
prefect backend server
prefect server start
Environment
- Ubuntu 20.04
- Fresh virtualenv with only prefect installed, versions 0.14.7+ fail, versions <0.14.7 work as expected.
Solution
I tested prefect versions 0.14.0 to 0.14.9 and observed that the bug was introduced in release 0.14.7 in src/prefect/cli/docker-compose.yml.
depends_on should be an array (starting by hyphen “-”) as stated in the error logs produced when running prefect server start, and the diff shows that hyphens were removed from depends_on options in 0.14.7.
See for instance : https://github.com/PrefectHQ/prefect/pull/4041/files#diff-110ca811f8b6466bd56ed8607c3a94de6925d89f71acedafd1ee71c166ab831dR54
–> Put hyphens back in depends_on options of the docker-compose.yaml 😃
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 18 (8 by maintainers)
Hi! Thanks for the report. This is the same issue as https://github.com/PrefectHQ/prefect/issues/4135 and is resolved by upgrading your docker-compose version. Perhaps we should switch to the old syntax to support a broader range of docker-compose versions, although upgrading docker-compose looks to be a simple fix. I’m not quite sure what change in docker-compose did this so I’ll have to test a bunch of versions to see what the deal is.
Disabling docker-compose v2 appears to have solved the issue:
I am strongly in favor of reverting the depends_on change until major distributions release newer versions of docker-compose.
I agree that the docker-compose dependency should not be part of prefect. I would even go further and consider that generating a docker-compose file and ensuring the containers are running is a complete separate concern that is more “infrastructure” than “development” and thus is likely to be handled by different people, with different constraints. That is to say, deploying and managing a prefect server is something a devops team would handle, and prefect developers would typically not care much as to how the server is managed.
With this philosophy, I do find it more natural that prefect servers can be fully installed and administered using readily available OS packages, and not require a separate virtualenv.
For instance, I would typically like to use the docker-compose OS package (rpm/deb) of my distribution and manage systemd services to ensure the containers are up and running. Having to rely on a docker compose that is not available yet on main Linux distributions is annoying, it requires the deployment of a custom, system level virtualenv containing the latest docker-compose and use this in systemd services.
TLDR: you’re making my life harder by requiring a docker-compose that is not available on centos/debian/ubuntu. My puppet/ansible/chef code to deploy and manage the containers for prefect server now has to deal with systemd units that reference some ad-hoc virtualenv with an up to date docker-compose.