dask-jobqueue: Incorrect description for env_extra for HTCondorCluster
Hi,
The description for env_extra in HTCondorCluster is not correct: the job that HTCondorCluster creates calls dask-worker directly instead of through a bash wrapper script, so you cannot put arbitrary shell commands into env_extra.
The interface supports environment variables as key=value pairs, which will be inserted into dask-worker’s environment (via the “Environment” attribute in the submit file). (For consistency, you can write export foo=bar but the word “export” will be ignored.)
This is also important to keep in mind with regards to #323; renaming env_extra to job_script_extra or similar would be even more inaccurate (for the HTCondor case anyway).
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (20 by maintainers)
Commits related to this issue
- fix behaviour of `env_extra` for HTCondor Before only environment variables were considered using HTCondor's `Environment =` feature, whereas the parameter description generally says "Other commands ... — committed to jolange/dask-jobqueue by jolange 2 years ago
- Fix behaviour of `env_extra` for HTCondor (#563) * fix behaviour of `env_extra` for HTCondor Before only environment variables were considered using HTCondor's `Environment =` feature, whereas th... — committed to dask/dask-jobqueue by jolange 2 years ago
- Fix behaviour of `env_extra` for HTCondor (#563) * fix behaviour of `env_extra` for HTCondor Before only environment variables were considered using HTCondor's `Environment =` feature, whereas th... — committed to ikabadzhov/dask-jobqueue by jolange 2 years ago
- Use `--nworkers` in stead of deprecated `--nprocs` in the generated job scripts (#560) * Use `--nworkers` in stead of deprecated `--nprocs` in the generated job scripts Fix https://github.com/dask... — committed to dask/dask-jobqueue by ikabadzhov 2 years ago
Ok, I made a proposal in #563. I’d suggest to start the renaming (#323) after we agreed on the implementation and merged it. (I can also do that if you like.)
😆
So by now I would prefer the in-lining solution to an extra temporary script. But maybe someone else has a good idea how to realize this.
I’d say go for the in-lining solution for the moment!
@jolange I think in this case you have already a wrapper script, it’s just inlined into the Arguments line 😃
Aside from the quoting problems that you mentioned, having a hard-to-read Arguments line just makes problems harder to debug, so I would prefer having the wrapper script be a separate file.