pipdeptree: Doesn't work with extras_require and tests_require

I was a bit surprised, that pipdeptree can not handle either extra, or test dependencies. See for example: Expected:

$ python -mpipdeptree
pipdeptree==0.13.0
  - pip [required: >=6.0.0, installed: 18.0]
pywikibot==3.0.dev0
  - crontab==0.22.2
  - flickrapi==2.4.0
    - requests [required: >=2.2.1, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-oauthlib [required: >=0.4.0, installed: 1.0.0]
      - oauthlib [required: >=0.6.2, installed: 2.1.0]
      - requests [required: >=2.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-toolbelt [required: >=0.3.1, installed: 0.8.0]
      - requests [required: >=2.0.1,<3.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: >=1.5.2, installed: 1.11.0]
  - google==2.0.1
    - beautifulsoup4 [required: Any, installed: 4.6.3]
  - irc==16.4
    - jaraco.collections [required: Any, installed: 1.5.3]
      - jaraco.classes [required: Any, installed: 1.5]
        - six [required: Any, installed: 1.11.0]
      - jaraco.text [required: Any, installed: 1.10.1]
        - jaraco.functools [required: Any, installed: 1.20]
          - more-itertools [required: Any, installed: 4.3.0]
            - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - six [required: >=1.7.0, installed: 1.11.0]
    - jaraco.functools [required: >=1.20, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - jaraco.itertools [required: >=1.8, installed: 2.5.2]
      - inflect [required: Any, installed: 1.0.1]
      - more-itertools [required: >=4.0.0, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - six [required: Any, installed: 1.11.0]
    - jaraco.logging [required: Any, installed: 1.5.2]
      - six [required: Any, installed: 1.11.0]
      - tempora [required: Any, installed: 1.13]
        - jaraco.functools [required: >=1.20, installed: 1.20]
          - more-itertools [required: Any, installed: 4.3.0]
            - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
        - pytz [required: Any, installed: 2018.5]
        - six [required: Any, installed: 1.11.0]
    - jaraco.stream [required: Any, installed: 1.2]
      - six [required: Any, installed: 1.11.0]
    - jaraco.text [required: Any, installed: 1.10.1]
      - jaraco.collections [required: Any, installed: 1.5.3]
        - jaraco.classes [required: Any, installed: 1.5]
          - six [required: Any, installed: 1.11.0]
        - six [required: >=1.7.0, installed: 1.11.0]
      - jaraco.functools [required: Any, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - more-itertools [required: Any, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.5]
    - six [required: Any, installed: 1.11.0]
    - tempora [required: >=1.6, installed: 1.13]
      - jaraco.functools [required: >=1.20, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - pytz [required: Any, installed: 2018.5]
      - six [required: Any, installed: 1.11.0]
  - memento-client==0.6.1
    - requests [required: >=2.7.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - mwoauth==0.3.2
    - oauthlib [required: Any, installed: 2.1.0]
    - PyJWT [required: >=1.0.1,<2.0.0, installed: 1.6.4]
    - requests [required: Any, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - requests-oauthlib [required: Any, installed: 1.0.0]
      - oauthlib [required: >=0.6.2, installed: 2.1.0]
      - requests [required: >=2.0.0, installed: 2.19.1]
        - chardet [required: >=3.0.2, installed: 3.0.4]
        - idna [required: >=2.5, installed: 2.7]
        - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: Any, installed: 1.11.0]
  - mwparserfromhell==0.5.1
  - Pillow==5.3.0
  - pycountry==18.5.26
  - pydot==1.2.4
    - pyparsing [required: >=2.1.4, installed: 2.2.2]
  - PyMySQL==0.9.2
    - cryptography [required: Any, installed: 2.3.1]
      - asn1crypto [required: >=0.21.0, installed: 0.24.0]
      - cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
        - pycparser [required: Any, installed: 2.19]
      - idna [required: >=2.1, installed: 2.7]
      - six [required: >=1.4.1, installed: 1.11.0]
  - python-stdnum==1.9
  - requests [required: >=2.9,!=2.18.2, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - sseclient==0.0.19
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
    - six [required: Any, installed: 1.11.0]

Current:

$ python -mpipdeptree
crontab==0.22.2
flickrapi==2.4.0
  - requests [required: >=2.2.1, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-oauthlib [required: >=0.4.0, installed: 1.0.0]
    - oauthlib [required: >=0.6.2, installed: 2.1.0]
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-toolbelt [required: >=0.3.1, installed: 0.8.0]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: >=1.5.2, installed: 1.11.0]
google==2.0.1
  - beautifulsoup4 [required: Any, installed: 4.6.3]
irc==16.4
  - jaraco.collections [required: Any, installed: 1.5.3]
    - jaraco.classes [required: Any, installed: 1.5]
      - six [required: Any, installed: 1.11.0]
    - jaraco.text [required: Any, installed: 1.10.1]
      - jaraco.functools [required: Any, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - six [required: >=1.7.0, installed: 1.11.0]
  - jaraco.functools [required: >=1.20, installed: 1.20]
    - more-itertools [required: Any, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - jaraco.itertools [required: >=1.8, installed: 2.5.2]
    - inflect [required: Any, installed: 1.0.1]
    - more-itertools [required: >=4.0.0, installed: 4.3.0]
      - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - six [required: Any, installed: 1.11.0]
  - jaraco.logging [required: Any, installed: 1.5.2]
    - six [required: Any, installed: 1.11.0]
    - tempora [required: Any, installed: 1.13]
      - jaraco.functools [required: >=1.20, installed: 1.20]
        - more-itertools [required: Any, installed: 4.3.0]
          - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
      - pytz [required: Any, installed: 2018.5]
      - six [required: Any, installed: 1.11.0]
  - jaraco.stream [required: Any, installed: 1.2]
    - six [required: Any, installed: 1.11.0]
  - jaraco.text [required: Any, installed: 1.10.1]
    - jaraco.collections [required: Any, installed: 1.5.3]
      - jaraco.classes [required: Any, installed: 1.5]
        - six [required: Any, installed: 1.11.0]
      - six [required: >=1.7.0, installed: 1.11.0]
    - jaraco.functools [required: Any, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - more-itertools [required: Any, installed: 4.3.0]
    - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
  - pytz [required: Any, installed: 2018.5]
  - six [required: Any, installed: 1.11.0]
  - tempora [required: >=1.6, installed: 1.13]
    - jaraco.functools [required: >=1.20, installed: 1.20]
      - more-itertools [required: Any, installed: 4.3.0]
        - six [required: >=1.0.0,<2.0.0, installed: 1.11.0]
    - pytz [required: Any, installed: 2018.5]
    - six [required: Any, installed: 1.11.0]
memento-client==0.6.1
  - requests [required: >=2.7.0, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
mwoauth==0.3.2
  - oauthlib [required: Any, installed: 2.1.0]
  - PyJWT [required: >=1.0.1,<2.0.0, installed: 1.6.4]
  - requests [required: Any, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - requests-oauthlib [required: Any, installed: 1.0.0]
    - oauthlib [required: >=0.6.2, installed: 2.1.0]
    - requests [required: >=2.0.0, installed: 2.19.1]
      - chardet [required: >=3.0.2, installed: 3.0.4]
      - idna [required: >=2.5, installed: 2.7]
      - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: Any, installed: 1.11.0]
mwparserfromhell==0.5.1
Pillow==5.3.0
pipdeptree==0.13.0
  - pip [required: >=6.0.0, installed: 18.0]
pycountry==18.5.26
pydot==1.2.4
  - pyparsing [required: >=2.1.4, installed: 2.2.2]
PyMySQL==0.9.2
  - cryptography [required: Any, installed: 2.3.1]
    - asn1crypto [required: >=0.21.0, installed: 0.24.0]
    - cffi [required: >=1.7,!=1.11.3, installed: 1.11.5]
      - pycparser [required: Any, installed: 2.19]
    - idna [required: >=2.1, installed: 2.7]
    - six [required: >=1.4.1, installed: 1.11.0]
python-stdnum==1.9
pywikibot==3.0.dev0
  - requests [required: >=2.9,!=2.18.2, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
sseclient==0.0.19
  - requests [required: >=2.0.0, installed: 2.19.1]
    - chardet [required: >=3.0.2, installed: 3.0.4]
    - idna [required: >=2.5, installed: 2.7]
    - urllib3 [required: >=1.21.1, installed: 1.23]
  - six [required: Any, installed: 1.11.0]

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 10
Comments: 15 (6 by maintainers)

Commits related to this issue

Manually add lunr --> nltk Workaround of https://github.com/tox-dev/pipdeptree/issues/107 — committed to geotribu/cli by Guts a year ago

Most upvoted comments

@naiquevin Note that since Python 3.8 you can use importlib.metadata to get this “extra” info easily:

>>> from importlib.metadata import requires
>>> requires("jira")
['defusedxml', 'oauthlib[signedtoken] (>=1.0.0)', 'pbr (>=3.0.0)', 'requests-oauthlib (>=0.6.1)', 'requests (>=2.10.0)', 'requests-toolbelt', 'setuptools (>=20.10.1)', 'six (>=1.10.0)', "argparse; (python_version<'2.7')", "requests-futures (>=0.9.7); extra == 'async'", 'ipython (<6.0.0,>=4.0.0); python_version < "3.3" and extra == \'cli\'', 'ipython (>=4.0.0); python_version >= "3.3" and extra == \'cli\'', "filemagic (>=1.6); extra == 'opt'", "PyJWT; extra == 'opt'", "requests-jwt; extra == 'opt'", "requests-kerberos; extra == 'opt'"]
>>> requires("oauthlib")
["cryptography ; extra == 'rsa'", "blinker ; extra == 'signals'", "cryptography ; extra == 'signedtoken'", "pyjwt (>=1.0.0) ; extra == 'signedtoken'"]

@AWhetter pip._internal should not be imported. It’s private API.

wimglenn on Dec 11, 2019

Hi, I didn’t get what you exactly mean by “handle” extra/test dependencies. It would be helpful if you could please add just the line diff between expected and actual output.

The example shows 20 packages. But 18 of them were installed using pip install -e pywikibot[extras] (or pip install -r requirements.txt) (as extra dependencies of pywikibot) and therefore should be marked as dependencies of pywikibot. They are not. They are marked as explicitly installed packages, which they are not. extras_require and test_require are parts of pip specification, both containing dependencies similar to install_require (which contains the dependencies pipdeptree can work with).

This issue is a huge limitation as it does not mark dependencies as dependencies and therefore the tree is not correct.

dvorapa on Oct 9, 2018

Do you plan to fix this issue?

ralphie0112358 on Mar 1, 2022

You can use importlib_metadata, the backport on PyPI

flying-sheep on Nov 8, 2021

We can use graphviz’ shape=record for this. One real life example (simplified):

digraph structs {
    node [shape=record]
    edge [color="#00000077"]
    
    api    [label="{ <base> api    | { <test> test | <dev> dev | <testing> testing }}"]
    server [label="{ <base> server | { <test> test | <dev> dev | <testing> testing }}"]
    utils  [label="{ <base> utils  | { <test> test | <dev> dev }}"]
    pytest [label="<base> pytest"]

    api:base -> utils:base
    server:base -> utils:base

    api:testing -> server:testing
    utils:test -> pytest:base
    api:test -> pytest:base
    server:test -> pytest:base
    server:testing -> pytest:base
}

grafik

Here, each packages’ tests depend on pytest, whereas both api and server have a extra called testing (containing test utils that other packages can use).

pipdeptree just ignores that api has an optional dependency on server, leaving out the actually important information I would have needed to debug a nasty dependency issue. I would have needed an output like this:

$ pipdeptree -p api[testing] --graph-output=dot | xdot /dev/stdin

grafik

flying-sheep on Aug 17, 2021

The DistInfoDistribution.requires() (the class that pip returns from get_installed_distributions()) takes an extras argument to specify which extras to query for requirements.

from pip._internal.utils.misc import get_installed_distributions
help(get_installed_distributions()[13].requires)

Each Requirement object returned from requires() has an extras that says which extras the requirement comes from.

get_installed_distributions()[13].requires()[2].extras

It should be possible to use these to store extras information in the tree somehow.

For a smaller test case, installing jira gives the following deptree:

cryptography==2.7
  - asn1crypto [required: >=0.21.0, installed: 0.24.0]
  - cffi [required: >=1.8,!=1.11.3, installed: 1.12.3]
    - pycparser [required: Any, installed: 2.19]
  - enum34 [required: Any, installed: 1.1.6]
  - ipaddress [required: Any, installed: 1.0.22]
  - six [required: >=1.4.1, installed: 1.12.0]
jira==2.0.0
  - defusedxml [required: Any, installed: 0.6.0]
  - oauthlib [required: >=1.0.0, installed: 3.0.2]
  - pbr [required: >=3.0.0, installed: 5.4.1]
  - requests [required: >=2.10.0, installed: 2.22.0]
    - certifi [required: >=2017.4.17, installed: 2019.6.16]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.9, installed: 2.8]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - requests-oauthlib [required: >=0.6.1, installed: 1.2.0]
    - oauthlib [required: >=3.0.0, installed: 3.0.2]
    - requests [required: >=2.0.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - requests-toolbelt [required: Any, installed: 0.9.1]
    - requests [required: >=2.0.1,<3.0.0, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.6.16]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.3]
  - setuptools [required: >=20.10.1, installed: 41.0.1]
  - six [required: >=1.10.0, installed: 1.12.0]
pipdeptree==0.13.2
  - pip [required: >=6.0.0, installed: 19.2.1]
PyJWT==1.7.1
wheel==0.33.4

Jira actually depends on oauthlib[signedtoken] (ie oauthlib with the signedtoken extra), which depends on cryptography and PyJWT. Because pipdeptree does not detect either as a dependency of the oauthlib extra, they are incorrectly listed as top level packages.

AWhetter on Jul 26, 2019