pytorch-lightning: Pip and conda install pydantic 2.0.2 by default, that causes ImportError

Bug description

I am developing library that requires lightning dependency. I am trying to build it using conda-build, but it crashes on testing with error ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main'. It also crashes with the same error when I tried to test it with tox in appveyor in python 3.8, 3.9, 3.10 and 3.11 on both windows and ubuntu.

I also tried to create new clean venv in pycharm, install lightning via pip and import lightning shows the same error.

Looks like pip and conda install pydantic 2.0.2 by default, but in this version ‘ModelMetaclass’ is not located in ‘pydantic.main’ anymore.

Manual installation of pydantic <2.0 or adding requirement pydantic <2.0 to requirements in my setup.py and meta.yaml fixes the problem.

UPD: also I found out that in wheel from pypi in METADATA file pydantic dependency is Requires-Dist: pydantic (<4.0,>=1.7.4), in tar.gz from pypi in requirements/base.txt it is pydantic<3.0,>=1.7.4 and in requirements/app/base.txt it is pydantic >=1.7.4, <2.0.0. conda search lightning --info also shows dependency - pydantic <4.0,>=1.7.4. I think it was supposed to be <2.0, not <4.0?

What version are you seeing the problem on?

v2.0

How to reproduce the bug

1. Create new python venv
2. pip install lightning
3. import lightning

Error messages and logs

ImportError while importing test module 'C:\projects\remote-sensing-processor\tests\test_landsat.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Python311-x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_landsat.py:4: in <module>
    import remote_sensing_processor as rsp
..\.tox\python3.11\Lib\site-packages\remote_sensing_processor\__init__.py:21: in <module>
    from remote_sensing_processor import segmentation
..\.tox\python3.11\Lib\site-packages\remote_sensing_processor\segmentation\__init__.py:3: in <module>
    from remote_sensing_processor.segmentation.segmentation import segmentation_train, segmentation_test
..\.tox\python3.11\Lib\site-packages\remote_sensing_processor\segmentation\segmentation.py:12: in <module>
    import lightning as l
..\.tox\python3.11\Lib\site-packages\lightning\__init__.py:18: in <module>
    from lightning.app import storage  # noqa: E402
..\.tox\python3.11\Lib\site-packages\lightning\app\__init__.py:24: in <module>
    from lightning.app import components  # noqa: E402, F401
..\.tox\python3.11\Lib\site-packages\lightning\app\components\__init__.py:1: in <module>
    from lightning.app.components.database.client import DatabaseClient
..\.tox\python3.11\Lib\site-packages\lightning\app\components\database\__init__.py:1: in <module>
    from lightning.app.components.database.client import DatabaseClient
..\.tox\python3.11\Lib\site-packages\lightning\app\components\database\client.py:22: in <module>
    from lightning.app.components.database.utilities import _GeneralModel
..\.tox\python3.11\Lib\site-packages\lightning\app\components\database\utilities.py:23: in <module>
    from pydantic.main import ModelMetaclass
E   ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main' (C:\projects\remote-sensing-processor\.tox\python3.11\Lib\site-packages\pydantic\main.py)

Environment

pip list after pip install lightning in clean venv

Package             Version
------------------- --------
aiohttp             3.8.4
aiosignal           1.3.1
annotated-types     0.5.0
ansicon             1.89.0
anyio               3.7.1
arrow               1.2.3
async-timeout       4.0.2
attrs               23.1.0
beautifulsoup4      4.12.2
blessed             1.20.0
certifi             2023.5.7
charset-normalizer  3.2.0
click               8.1.4
colorama            0.4.6
croniter            1.3.15
dateutils           0.6.12
deepdiff            6.3.1
exceptiongroup      1.1.2
fastapi             0.100.0
filelock            3.12.2
frozenlist          1.3.3
fsspec              2023.6.0
h11                 0.14.0
idna                3.4
inquirer            3.1.3
itsdangerous        2.1.2
Jinja2              3.1.2
jinxed              1.2.0
lightning           2.0.4
lightning-cloud     0.5.37
lightning-utilities 0.9.0
markdown-it-py      3.0.0
MarkupSafe          2.1.3
mdurl               0.1.2
mpmath              1.3.0
multidict           6.0.4
networkx            3.1
numpy               1.25.0
ordered-set         4.1.0
packaging           23.1
pip                 22.1.2
psutil              5.9.5
pydantic            2.0.2
pydantic_core       2.1.2
Pygments            2.15.1
PyJWT               2.7.0
python-dateutil     2.8.2
python-editor       1.0.4
python-multipart    0.0.6
pytorch-lightning   2.0.4
pytz                2023.3
PyYAML              6.0
readchar            4.0.5
requests            2.31.0
rich                13.4.2
setuptools          62.3.3
six                 1.16.0
sniffio             1.3.0
soupsieve           2.4.1
starlette           0.27.0
starsessions        1.3.0
sympy               1.12
torch               2.0.1
torchmetrics        1.0.0
tqdm                4.65.0
traitlets           5.9.0
typing_extensions   4.7.1
urllib3             2.0.3
uvicorn             0.22.0
wcwidth             0.2.6
websocket-client    1.6.1
websockets          11.0.3
wheel               0.37.1
yarl                1.9.2

setup.py with requirements

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

with open("src/remote_sensing_processor/__init__.py") as f:
    for line in f:
        if line.find("__version__") >= 0:
            version = line.split("=")[1].strip()
            version = version.strip('"')
            version = version.strip("'")
            continue

setuptools.setup(
	name = 'remote-sensing-processor',
	version = version,
	author = 'Mikhail Moskovchenko',
	author_email = 'moskovchenkomike@gmail.com',
	description = 'RSP is a tool for geospatial raster data processing',
	long_description = long_description,
	long_description_content_type = 'text/markdown',
	url = 'https://github.com/simonreise/remote-sensing-processor',
	project_urls = {
		'Bug Tracker': 'https://github.com/simonreise/remote-sensing-processor/issues',
        'Source': 'https://github.com/simonreise/remote-sensing-processor',
        'Documentation': 'https://remote-sensing-processor.readthedocs.io'
	},
	classifiers = [
		'Programming Language :: Python :: 3',
		'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
		'Operating System :: OS Independent',
		'Environment :: GPU :: NVIDIA CUDA',
		'Topic :: Scientific/Engineering :: GIS'
	],
    keywords = 'remote sensing, landsat, sentinel, gdal, rasterio',
	package_dir = {"": "src"},
    packages = setuptools.find_packages(where="src"),
    setup_requires = ['cython', 'numpy>=1.17'],
    install_requires = ['numpy>=1.17', 'h5py', 'torch>=1.10', 'torchvision>=0.10', 'lightning', 'tensorboard', 'transformers', 'timm', 'scikit-learn', 'scikit-image', 'rasterio', 'pyproj', 'geopandas'],
    python_requires = ">=3.8",
    include_package_data = True
)

meta.yaml for conda-build

{% set data = load_setup_py_data() %}

package:
  name: remote-sensing-processor
  version: {{ data.get('version') }}

source:
  path: ./..

requirements:
  build:
    - python {{ python }}
    - cython
    - numpy >=1.17
  host:
    - geopandas
    - h5py
    - lightning
    - numpy >=1.17
    - pip
    - pyproj
    - python {{ python }}
    - pytorch >=1.10
    - rasterio
    - scikit-image
    - scikit-learn
    - tensorboard
    - timm
    - torchvision
    - transformers
  run:
    - geopandas
    - h5py
    - lightning
    - numpy >=1.17
    - pyproj
    - python {{ python }}
    - pytorch >=1.10
    - rasterio
    - scikit-image
    - scikit-learn
    - tensorboard
    - timm
    - torchvision
    - transformers


build:
  noarch: python

test:
  imports:
    - remote_sensing_processor
    - remote_sensing_processor.common
    - remote_sensing_processor.imagery_types
    - remote_sensing_processor.indices
    - remote_sensing_processor.landsat
    - remote_sensing_processor.mosaic
    - remote_sensing_processor.segmentation
    - remote_sensing_processor.sentinel2
    - remote_sensing_processor.sentinel2.sen2cor
    - remote_sensing_processor.sentinel2.sentinel_postprocessing
    - remote_sensing_processor.sentinel2.superres
    - remote_sensing_processor.sentinel2.superres.weights
    - remote_sensing_processor.unzip


about:
  home: "https://github.com/simonreise/remote-sensing-processor"
  license: "GNU General Public v3 (GPLv3)"
  license_family: GPL3
  license_file:
  summary: "RSP is a tool for geospatial raster data processing"
  doc_url: "https://remote-sensing-processor.readthedocs.io"
  dev_url:

appveyor.yml

environment:
  matrix:
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.8"
      PYTHONPATH: "C:\\Python38-x64"
      TOXENV: python3.8
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.9"
      PYTHONPATH: "C:\\Python39-x64"
      TOXENV: python3.9
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.10"
      PYTHONPATH: "C:\\Python310-x64"
      TOXENV: python3.10
    - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
      PYTHON: "3.11"
      PYTHONPATH: "C:\\Python311-x64"
      TOXENV: python3.11
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.8"
      TOXENV: python3.8
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.9"
      TOXENV: python3.9
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.10"
      TOXENV: python3.10
    - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
      PYTHON: "3.11"
      TOXENV: python3.11
stack: python %PYTHON%

build: false

for:
-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.8"
        PYTHONPATH: "C:\\Python38-x64"
        TOXENV: python3.8 
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.1.10.1/GDAL-3.6.2-cp38-cp38-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.1.10.1/rasterio-1.3.4-cp38-cp38-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.1.10.1/Fiona-1.8.22-cp38-cp38-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"    

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.9"
        PYTHONPATH: "C:\\Python39-x64"
        TOXENV: python3.9
    
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/GDAL-3.6.4-cp39-cp39-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/rasterio-1.3.6-cp39-cp39-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/Fiona-1.9.3-cp39-cp39-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"
    
-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.10"
        PYTHONPATH: "C:\\Python310-x64"
        TOXENV: python3.10
    
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/GDAL-3.6.4-cp310-cp310-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/rasterio-1.3.6-cp310-cp310-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/Fiona-1.9.3-cp310-cp310-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022
        PYTHON: "3.11"
        PYTHONPATH: "C:\\Python311-x64"
        TOXENV: python3.11
    
  install:
    - "SET PATH=%PYTHONPATH%;%PYTHONPATH%\\Scripts;%PATH%"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/GDAL-3.6.4-cp311-cp311-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/rasterio-1.3.6-cp311-cp311-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.4.22/Fiona-1.9.3-cp311-cp311-win_amd64.whl"
    - "%PYTHONPATH%\\python.exe -m pip install tox"
  test_script:
    - "%PYTHONPATH%\\python.exe -m tox"

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.8"
        TOXENV: python3.8
  install: 
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.9"
        TOXENV: python3.9
  install: 
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.10"
        TOXENV: python3.10
  install:
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

-
  matrix:
    only:
      - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
        PYTHON: "3.11"
        TOXENV: python3.11
  install:
    - sh: sudo add-apt-repository ppa:ubuntugis/ppa
    - sh: sudo apt-get update
    - sh: sudo apt-get -y install python-numpy gdal-bin libgdal-dev
    - sh: sudo apt-get -y install python3-pip
    - pip3 install tox
  test_script:
    - sh: python3 -m tox

More info

No response

cc @tchaton

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

@function2-llx, I’ve pinned pydantic<2.0 in the lightning=2.0.5 conda-forge package at https://github.com/conda-forge/lightning-feedstock/pull/8, maybe try installing again and see if it works for you.

+3
weiji14 on Jul 17, 2023

I have similar issue, import pytorch_lightning works well However from lightning.pytorch.utilities import rank_zero_only failed with the following error ImportError: cannot import name 'ModelMetaclass' from 'pydantic.main' pydantic 2.3.0 pyhd8ed1ab_0 defaults pydantic-core 2.6.3 py310hcb5633a_0 defaults

pytorch-lightning 2.0.8 pyhd8ed1ab_0 defaults

+2
ItamarKanter on Sep 11, 2023

Seems to be fixed by #18022

+2
AnezeR on Jul 8, 2023

I believe the correct fix is here: https://github.com/Lightning-AI/lightning/pull/18030 @Borda This commit was missed in the 2.0.5 release.

@function2-llx The solution is to downgrade pydantic to < 2.0 until Lightning 2.0.6 is released.

+1
awaelchli on Jul 12, 2023
Read more comments on GitHub
← pytorch-lightning: overfit_batches doesn't work
pytorch-lightning: pl+wandb: Hanging during "cleaning up ddp environment" when using DDPSpawnPlugin + WandbLogger →