pre-commit: pre-commit doesn't keep current project/virtualenv in sys.path

I’m trying to get Prospector ( https://github.com/landscapeio/prospector ) to run as a pre-commit hook. By default, Prospector will run pylint, including pylint’s import checker.

I was getting confused, because I kept getting output that looks like:

test_mailing_lists (<projectname>/tests/services/platform/email/test_mailing_lists.py):
    L8:0 None: pylint - F0401
    Unable to import '<projectname>.services.platform.email.mailing_lists'

Even though directly running Prospector in the project (as opposed to via pre-commit) doesn’t raise these errors.

When I print sys.path from inside a Python script run by pre-commit as a hook, I get:

['/Users/gkisel/.pre-commit/repodfiUn9/py_env/bin',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python27.zip',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/plat-darwin',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/plat-mac',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/Extras/lib/python',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/lib-tk',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/lib-old',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/site-packages',
 '/Users/gkisel/.pre-commit/repodfiUn9/py_env/lib/python2.7/site-packages/astroid/brain']

Note that this doesn’t appear to include my project directory or my virtualenv path. I think this is why pylint (within Prospector) is unable to successfully run those import statements.

Is this behavior intentional? Am I doing something incorrectly? Is there a way I can get pre-commit to preserve my sys.path when running Python hooks?

For reference, see https://github.com/guykisel/prospector-mirror

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 24 (11 by maintainers)

Most upvoted comments

Another tool that is affected by this problem is mypy, it really needs stuff from the project’s venv to work properly unless one pretty much completely ignores errors resulting from imports.

I think the system workaround isn’t really applicable for cases where for example a git GUI tool (such as GitHub Desktop) is launched and stuff committed to a pre-commit enabled working dir; there’s likely no opportunity to activate the project venv there.

Maybe a script approach that would activate the venv would work, dunno. But a problem there is that the way people use venvs differ; some prefer plain python -m venvs, others pyenv virtualenvs, the dirs where the venvs are set up vary, so the script would not be able to activate the venv properly in all cases, and forcing people to use virtualenvs some specific way would be kind of meh.

So it would be great if pre-commit would offer some more assistance in coping with these scenarios. Don’t know what exactly would that be. Maybe e.g. some option that would “record” the currently active virtualenv settings at hook install time and use them later to construct the same environment when running the installed hook, dunno.

Just in case:

I had next hook:

  - repo: local
    hooks:
      - id: pytest
        name: pytest
        entry: poetry run pytest --
        language: python
        types: [python]
        files: ^tests\/.*test_.*\.py$
        pass_filenames: false  # or use with: require_serial

I use poetry. And on each changes of tests, it used different python.

Base on comment @asottile , I changed language: python to language: system and everything started works.

there’s no point to doing that in pre-commit’s configuration – I was only suggesting you try that in your shell to establish an amount of sanity / correct expectations for what pre-commit is going to do (while eliminating the cache invalidation problem of your shell)

right – I was mostly explaining why in your screenshot above you see a different which result inside / outside of pre-commit. because the whichs outside are using your shell’s hash cache despite not actually being available.

For language: system you should be able to run the same command in and outside of pre-commit and receive the same results – if you can show that not happening then there’s a bug (though a report that involves which or shell builtins should run hash -r before invoking those commands). Alternatively you can use python3 -c 'import shutil; print(shutil.which("python"))' etc.


an aside, one thing I noticed about the screenshot above is the which hooks show up as Failed – however that seems impossible since which produced a result and exited zero – was your configuration something different from that?