pydantic: Import slowdown after v1

After upgrading to v1, I’ve noticed that it takes almost double the time to import Pydantic. Observe:

$ python3 -m pip install -U 'pydantic<1'
[...]
Successfully installed pydantic-0.32.2

$ seq 2 | xargs -I -- python3 -X importtime -c '
  import pydantic
  print("pydantic version:", pydantic.VERSION)
  ' 2>&1 | begin
      head -n 1
      tail -n 2
  end
import time: self [us] | cumulative | imported package
import time:      1058 |      70730 | pydantic
pydantic version: 0.32.2

$ python3 -m pip install -U pydantic
[...]
Successfully installed pydantic-1.3

$ seq 2 | xargs -I -- python3 -X importtime -c '
  import pydantic
  print("pydantic version:", pydantic.VERSION)
  ' 2>&1 | begin
      head -n 1
      tail -n 2
  end
import time: self [us] | cumulative | imported package
import time:       802 |     128663 | pydantic
pydantic version: 1.3

v0.32.2 takes approximately 70ms to import on my MBP; v1.3 takes 130ms.

attrs for comparison takes 36ms:

$ seq 2 | xargs -I -- python3 -X importtime -c '
  import attr
  print("attrs version:", attr.__version__)
  ' 2>&1 | begin
      head -n 1
      tail -n 2
  end
import time: self [us] | cumulative | imported package
import time:       882 |      35659 | attr
attrs version: 19.3.0

I can understand that this might not be something you’d want to spend time on, depending on what your direction for Pydantic is; but slow warmups have been plaguing Python CLIs for a very long time. If you’d like to investigate this further, tuna can help you visualise your dependency tree. Here’s what it’s showing me for tuna (python3 -X importtime -c 'import pydantic' 2>| psub):

Screenshot_2019-12-24 tuna - var folders 18 r2nky_v137d3fwhtx0644m940000gn T psub WY0ulb7Mci

There is the unfortunate practice in Python of importing the entire package’s contents in __init__ and (unfortunately) once you’ve done that, there’s no going back - not without a breaking release anyway.

As always, thanks for all your work on Pydantic.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (16 by maintainers)

Commits related to this issue

Most upvoted comments

It appears to be caused, in no small part, by the pre-compilation of this particular regex with a range containing 262,057 characters:

_domain_ending = r'(?P<tld>\.[a-z]{2,63})?\.?'
_int_chunk = r'[_0-9a-\U00040000](?:[-_0-9a-\U00040000]{0,61}[_0-9a-\U00040000])?'
int_domain_regex = re.compile(fr'(?:{_int_chunk}\.)*?{_int_chunk}{_domain_ending}', re.IGNORECASE)

I think the re module might’ve not been optimised for this particular use case 😉