dvc: Virtualenv on macOS not working (fish)

Please provide information about your setup

DVC version: 0.20.1
Platform: macOS
method of installation: pip

It looks like the commands run by dvc are ignoring currently used virtualenv screenshot 2018-11-08 at 13 48 32

Steps to reproduce the issue:

$ mkdir dvc_test && cd dvc_test
$ git init
$ virtualenv -p python3.6 .env
$ source .env/bin/activate
$ pip install dvc
$ dvc init
$ which python
$ dvc run which python

The output of two last commands is different, which means, that python… comannd run by DVC is not taking into acount activated virtualenv. I’ve tested with virtualenv as well as with python venv.

I’ve done the same steps on linux - it works fine, so it’s seems that the issue apears only on macOS

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 30 (16 by maintainers)

Commits related to this issue

run: warn if using fish shell Fixes #1307 Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai> — committed to efiop/dvc by efiop 6 years ago
Use `--no-config` option in fish > 3.3 Prevent fish to reload config files in non-interactive mode. Disable warning when using a recent fish version. Fixes #1307 — committed to clamydo/dvc by clamydo 2 years ago

Most upvoted comments

I use fish + conda (miniforge).

I tried various approaches listed here, and I found only one approach to solve it : SHELL=/bin/bash dvc ...

To have a smoother process I leverage fish abbrs:

abbr bs SHELL=/bin/bash and then bs dvc ...

horaceg on Nov 18, 2021

@hanan-vian Do any of the workarounds described above work for you?

Should I give up fish just so I can use dvc?

Not at all, you just need to configure your setup to make it work.

By the way, if I run dvc run in a bash it will not help as my default is still fish.

That’s because your SHELL env var keeps pointing to fish. You could set it to bash or something else and it should work fine.

I guess, dvc config should include a shell settings or a conda env setting so that one can run in fish

Sounds too hackish. The proper solution is probably to make fish support an option (e.g. --norc) that will make it usable, but fish folks seem to be against it fish-shell/fish-shell#5394 . Please feel free to comment on that issue, maybe the pressure from actual users will make them reconsider it.

As I’ve mentioned previously, we could consider launching bash (or something else if available) if we detect fish, but again that is pretty hacky and might result in some unexpected behavior. It would be great if someone who uses fish could give that approach a try and see if that works for him or not. We would be glad to accept such a patch.

See my conda run hack.

ghost on May 31, 2020

I don’t think we should close this issue, as a warning message is not a solution. Should I give up fish just so I can use dvc? By the way, if I run dvc run in a bash it will not help as my default is still fish. I guess, dvc config should include a shell settings or a conda env setting so that one can run in fish

ghost on May 31, 2020

@svenski the way fish handles env vars is still quite error-prone, even without using conda. I’m afraid that if we remove that warning, users will not know where to start looking for a solution and we’ll simply get a lot of bug reports about this same thing. If that warning message is not too annoying for you, I would prefer to leave it as is for now.

efiop on Mar 21, 2019

This might help others that use fish and are confused by the warning message.

The recommended way of setting up conda is not to change the PATH env variable but to source a conda.sh (or conda.fish in this case). Here is a link to conda release notes from 2017-12 about this: https://github.com/conda/conda/blob/master/CHANGELOG.md#recommended-change-to-enable-conda-in-your-shell .

My fish.config has a line: source /opt/miniconda3/etc/fish/conf.d/conda.fish and I don’t see the issues mentioned in the original post. I think it is only an issue if you change the PATH.

Ideally, based on this, I’d prefer to remove the warning message since it not needed if you set up conda as recommended.

svenski on Mar 21, 2019

I have started an early attempt to fix this isse. See #8101 I am not happy about this solution. I haven’t found away avoiding spawning a command to find out about the shell version.

Perhaps somebody can chime in?

clamydo on Aug 7, 2022

@efiop Today, I started to hack something together. Just adding the option seems easy, however, checking if a fish version above 3.3 is installed and only then apply the option is a tad more fishy. I currently run fish --version in warn_if_fish and check for the version.

But I haven’t had time to figure out, how to cleanly adapt the _make_cmd function, without spawning a subprocess everytime. I hope I’ll find some time tomorrow.

I think it is necessary to check for the version to remain backwards-compatible, because fish 3.3.x and above isn’t necessarily wide-spread yet.

clamydo on Apr 22, 2022

Fish gained a --no-config option that could be used, when detecting fish (see https://github.com/fish-shell/fish-shell/pull/7921)

clamydo on Apr 21, 2022

Hi, I have a similar Issue.

The problem was that dvc run was restarting fish shell which activates condas base environment by default. This seems to be a fish shell limitation. My workaround for conda version 4.7 was:

conda init fish on the cli
add conda activate my_project_env to ~/.config/fish/config.fishafter the part added byconda init`

Now fish alway starts by activating this specific project environment, but i get to keep everything else.

kaiogu on Jan 24, 2020

@efiop , I don’t think it would impact my existing pipelines. I don’t use fish specific scripts, and if I did, I’d make sure there was a #! path/to/fish in them, so I think this is ok from impact perspective.

Nice to know that. I guess that is that way for most of fish users.

However, out of curiosity, can I ask why you want to do this? Are there any other issues with fish? The original issue in raised in this ticket can be avoided by setting up conda correctly (recommended) in fish (https://github.com/conda/conda/blob/master/CHANGELOG.md#recommended-change-to-enable-conda-in-your-shell).

Just something that came to my mind, so decided to ask for an opinion from fish users 😃 Glad to hear that properly setting up conda works for you!

So IMHO I don’t think this is issue with dvc, it’s with setting up conda in fish. TBH, I would opt for removing the warning unless of course there are other issues with using fish shell that are not mentioned in here.

It is not only about conda though, in the linked PR we’ve been running into other env-related issues and fish is especially prone to those because of the way it handles .fishrc. But it looks like except for conda, fish users are not doing anything crazy with .fishrc. So if it is working out for you right now as is and no one will complain about it, I agree that we should leave it as is for now. About the config option for disabling the warning, it is on our todo list, but I honestly can’t tell the ETA for it right now 🙁 That being said, if you would be interested in giving a shot implementing it, we will be extremely happy to help. 🙂

Thank you for the feedback!

efiop on Sep 30, 2019

@shcheklein - I don’t have enough understanding or context of dvc to meaningfully chip in on that decision.

However, from a (fish-shell) user point of view it would be nice, as I get a warning for each stage that I run now. But I also understand if you don’t want to complicate things too much. Ideally, it would be fixed for fish, even when changing the PATH, but I see that was attempted and abandoned.

I’m using dvc now for a project so I do plan to learn more about the internal in time, I’ll keep it in mind to try to help with fixing this issue if I have the chance.

svenski on Mar 21, 2019