azure-cli: Support PEP 420 -- Implicit Namespace Packages

As Python 2 has been deprecated, we should move away from the pkg_resources approach and use PEP 420 – Implicit Namespace Packages.

Importing pkg_resources will add a 100ms~200ms delay to CLI’s loading time.

$ time python -c "import pkg_resources"

real    0m0.153s
user    0m0.137s
sys     0m0.016s

time python -X importtime -m azure.cli version -h 2>perf.log
tuna perf.log

This is discussed at https://github.com/pypa/setuptools/issues/510 which affects many projects but still doesn’t have a proper fix.

Moreover, on Linux, the wrapper ~/env38/bin/az created by Easy Install also requires pkg_resources:

#!/home/admin/env38/bin/python
# EASY-INSTALL-DEV-SCRIPT: 'azure-cli==2.5.1','az'
__requires__ = 'azure-cli==2.5.1'
__import__('pkg_resources').require('azure-cli==2.5.1')
__file__ = '/home/admin/azure-cli/src/azure-cli/az'
with open(__file__) as f:
    exec(compile(f.read(), __file__, 'exec'))

This makes az installed via azdev even slower.

For comparison, directly run azure.cli:

$ time python -m azure.cli version -h

real    0m0.261s
user    0m0.238s
sys     0m0.024s

Run the Easy Install wrapper:

$ time python ~/env38/bin/az version -h

real    0m0.822s
user    0m0.779s
sys     0m0.044s

python -X importtime ~/env38/bin/az version -h 2>perf.log
tuna perf.log

In our Linux release, we package our own wrapper src/azure-cli/az, so it doesn’t suffer from the slowness of this Easy Install wrapper.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 19 (18 by maintainers)

Most upvoted comments

Yes, you can kill them now, their only purpose is Python 2, that you don’t handle anymore. Remove them all from everywhere.

lmazuel on May 16, 2020

@jiasli

On dev environment, you must NOT install any nspkg. For dev, there is no question that you want to leverage PEP420. The nspkg will indeed create trouble to import, at least if you venv is in editable mode.
Your production wheel must NOT contains azure\cli\__init__.py,
Your github repo should not contain any azure\cli\__init__.py. This file shouldn’t exist.

The debate we have with @zooba is solely about this line, and nothing else, the rest of the recommended guidelines still applies. Once everything else is fixed, and we’re down the decide this, we’ll settle the debate 😃. In the meantime, keep this line since it’s there, but there is many things to still fix it seems.

lmazuel on May 22, 2020

I don’t have time for 2 cents (plus I live in a country without a penny so I couldn’t give you 2 cents even if I wanted to 😉 ).

brettcannon on May 19, 2020

Side note @jiasli that I didn’t realize before @zooba comment, in real life CLI installation, you shouldn’t have the pkg_resources line anyway, since you’re not supposed to rely on nspkg packages today already. So I think you did you measurement on a dev install of the CLI. In order to get numbers that represents reality, I would encourage you do them on a real install.

If you do have pkg_resources on any real CLI installation, that’s a bug and that’s parallel to the discussion if we should use PEP420 (no __init__.py) or regular package (empty __init__.py file)

lmazuel on May 22, 2020

Since you don’t need Python 2, your situation is way simpler than it was when I created that issue, but might give you some insights. Feel free to contact me if you have questions on packaging.

lmazuel on May 5, 2020