tox: virtualenv is not recreated when deps change if '-r' is used

I’ll have to disagree - this is an issue I hear from various people, and it’s quite unexpected. I view it as “tox does caching without invalidating the cache correctly” essentially. The core does caching, so I think it really should do this correctly.

Just to give an update of this. There has been a significant discussion going on upstream to build backends to offer dependency information (and other metadata) without a build happening. Once this takes shape we’ll be able to implement the feature request here in a reliable fashion. I plan to push this through so should be addressed by the end of the year (my plan as of now).

I hope this bug does not endup closed before we have this fixed inside tox itself. We should not endup using external tools or hacking tox.ini in order to get around it (workaround which mainly disables the deps option in tox).

Yes, and it’s planned for tox 4 🤔

Using “-r requirements.txt” in tox is so common that I’m suprised this issue has remained unresolved for >4 years instead of being solved by a hotfix. When is tox 4 planned to be released?

It’s a fairly complicated issue, it’s more like adding a major feature than hotfix. We have an alpha version of it https://pypi.org/project/tox/4.0.0a2/, a first public release expected in a few months.

It is indeed https://github.com/jazzband/pip-tools

tox.ini:

[testenv]
deps =
    pip-tools == 1.8.1rc3
commands =
    pip-sync requirements.txt requirements-test.txt
    pyflakes project
    pytest {posargs}

pip-sync makes sure new dependencies or updated versions are installed in the virtualenv and removes old dependencies. I don’t duplicate dependencies in tox.ini, I never have to recreate the testenvs, my co-workers never have failures because requirements.txt changed. My deployment also uses pip-sync.

Article from the original developer of pip-tools: http://nvie.com/posts/better-package-management/

pip-tools very recently gained the ability to generate requirements.txt with pinned versions from dependencies in setup.py, so you can have a lib with loose requirements in setup.py and exact pinned versions in requirements-test.txt for your tox testing.

Original comment by @Merwok

This also happens if dependencies in setup.py (install_requires) change.

That’d be horrible IMHO, and keep a lot of projects (including mine) from using tox

This approach has been discussed before (probably multiple times by now?) in this thread. The main problem is that requirement files can include others via -r, which still wouldn’t be fixed by a “naive” approach. However, I’d argue that even a naive approach which doesn’t deal with requirement files including others would be a big benefit.

Yes, and it’s planned for tox 4 🤔

Removing the -r support would immediately break CI for a lot of people (almost nobody uses a pinned tox), and is a pretty important feature about tox, IMHO.

@gaborbernat How about adding a fixed-by-tox4 label and also locking the ticket? There are lots of other similar tickets which are solved by it and which will not be addressed by old version. I you give me triage rights, I could help with that.

Could tox hash any requirements.txt files that are used and cache that? Then it could know if the file had changed and recreate if necessary.

I’ve implemented a tox plugin to hook up venv-update to make it so tox -r is unnecessary. From my cursory testing, it seems to work pretty well:

• https://github.com/asottile/tox-pip-extensions

It’s a little rough around the edges:

• overrides install_command of all testenvs
• overrides installdeps step to preinstall the required bootstrap packages (in this case venv-update and its dependencies)
• Adds a pretest hook which --prunes extraneous dependencies

but it seems to work

This test demonstrates the acceptance criteria described in this ticket: https://github.com/asottile/tox-pip-extensions/blob/4d0ccaae3d68495d2d3e7c1bdb78102f0d6fafae/tests/tox_pip_extensions_test.py#L166-L182

we once had better support for requirements parsing but we had to revert it … i admit i never extensively used -r requyirements.txt myself and the docs still mark that feature as experimental http://tox.readthedocs.io/en/latest/example/basic.html?highlight=requirements#depending-on-requirements-txt – someone needs to step up to go for it more properly … meanwhile recommending to manually use “tox -r” for recreating an env is a kludge to get around tox’s ignorance.

Original comment by mhirota

What about recognizing the special case of deps = -r<something> and calculating the md5 based on the file contents? This would trigger recreate if anything in the requirements.txt changed… but I think that might be better than duplicating what is in pip or introducing a hard dependency on pip in tox.

I’ve wanted to take pip-faster install --prune, upstream it, and then use it for this purpose (probably as pip prune? similar to npm prune). But I haven’t really had the time or motivation (and sadly we don’t use tox at work so I can’t even float it there 😕). Until then, I’ve used tox-pip-extensions with the venv-update extension to work around this

+1 for @asottile’s tox-pip-extensions as a solution to this problem, though obviously it would still be nice to have it fixed in tox out of the box

Maybe we should remove the experimental mark then and document that the env recreation does not really work together with that feature (yet)?

Definitely. IMHO it is way past the “experimental” stage where it could be removed. Removing it would break tons of test suites everywhere.

disclaimer: I’m not (yet) familiar with tox’s internals and its design philosophy and AFAIK it is my first interaction with the project

Would a Make style, last-modified-time (dependencies vs. ‘sources’) based approach be an acceptable solution here? I.e.: in VirtualEnv.update, instead of using only self.envconfig.recreate, use (self.envconfig.recreate or should_recreate_based_on_last_modified_times()) (or store into a variable/method for brevity)? It could check the -r file as well as the setup.py in the root folder (*).

Granted, its performance wouldn’t be as good as a smart solution’s, but it would produce a correct venv

* and I know there can be weird corner cases with editable dependencies having their own setup.py files that can change, or the repo having multiple setup.py files, etc., but those could be fixed iteratively, by expanding the list of files to check for timestamps

edit: or is this suggestion better suited for #93?

@merwok that sounds interesting, but I have no idea what you are talking about - could you link to some documentation/blog/howto/whatever with more details about this approach? Thanks!

It would also make sense to order requirements before hashing, so change in order don’t result in an environment recreation.

Order is significant in requirements files (for now at least — Pip’s new resolver may change that).

@gaborbernat Really nice to hear you aim to nail it! On the other hand I realise that we are in March… so the end of the year is not as near as I would have wished for.

Ah I see, you’re misunderstanding how pip-sync works.

From https://github.com/jazzband/pip-tools#example-usage-for-pip-sync

Be careful: pip-sync is meant to be used only with a requirements.txt generated by pip-compile.

@obestwalter regarding the non working extras it’s actually my fault, the system where I was having the issue had tox 2.3.1 installed system-wise, and I cannot reproduce it with tox 2.5.0 or 2.7.0. Sorry about the misunderstanding.

As already stated, this doesn’t change the main issue here, that the dependencies are not updated. Anyway thanks for the tip.

Thanks for working on this!

Independently of the specific solution chosen, please consider to make it work also for keeping dependencies up to date when installed from setup.py, I think that this is also a fairly standard use case.

To be explicit I’m referring to the usage of deps = . to install the install_requires dependencies or deps = .[key_in_extras_require] to install the install_requires dependencies plus the ones defined in extras_require['key_in_extras_require'].

In the end I’m seeing this as a cache issue on tox side, given that there are at least three working syntaxes to install dependencies , of which only one express them explicitly, while the others get them indirectly and tox is not able to detect changes or available updates in the indirect cases.

maybe it would be better to deprecate the -r option

It’s still a useful shortcut to delete and re-run.

I think the -r that’s being talked about here is the one in deps = and not the command line one.

However wouldn’t it be more sensible to keep the -r deps capability but instead of reinventing the wheel just rely on pip-tools for the job? From a quick look at the source it seems as if using it as a library should be pretty easy (although maybe not officially supported?)

@obestwalter pip-tools can be used to keep a virtualenv exactly in sync with the requirements.

It’s really a great idea to use it in tox. Should have thought of that myself.

@obestwalter one approach (that we’re using internally, and will have a better solution for once 2.7.0 ships with my plugin changes 😄) is to use https://github.com/Yelp/venv-update – pip-faster has a --prune which will uninstall extraneous dependencies (and ensure the installed state is exactly what was on the cli)

Original comment by wbyoung

@wmyll6 thanks for the plugin. Hopefully some of that work can make its way into the core!