conan: [bug] Conan 2.X generates invalid transitive dependency info when building project with editable packages

Describe the bug

(not sure if this is actually a bug, might be my misunderstanding of the dependency model, but things worked just fine in 1.62) (also this could reeeallly benefit from a better title…)

We have a monorepo with quite a bunch (40+) libraries and executables under the same roof. We use Conan to work with them like this, essentially:

$ for P in project1, project2, ... : conan editable add ./$P

# in ./BigProject/conanfile.py:
name = "BigProject"
version = "editable"
self.requires("AnotherProject/editable")

# in ./BigProject/CMakeLists.txt:
add_subdirectory(../AnotherProject)
find_package(Eigen3 REQUIRED)
target_link_libraries(BigProject Eigen3::Eigen3 AnotherProject ...)

$ cd BigProject/build/
$ conan install ..
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=`pwd`/Conan/conan_toolchain.cmake
$ cmake --build .

All projects use the same path for generators directory; this is very likely suboptimal, but allows conan install to generate aaalll the package-Config.cmakes under one roof, and so the add_subdirectory(local) + find_package(thirdparty) approach works seamlessly. In Conan 1, that is.

With Conan 2 (tested with 2.10.2 in work repo and 2.11 in example repo) things get worse. With the dependency chain like BigProject -> Subproject -> Wrappers -> 3rdparty I now get cases where I can easily build Wrappers and Subproject on their own, but building the exact same targets as part of the BigProject fails, because the include directories for targets in 3rdparty from Conan are not set.


How to reproduce it

I made a repro repo here for convenience - but the issue can be reproduced without any CMake, only conanfiles are needed:

Conanfiles...

  • Start with a library that provides some very common functionality and maybe wrappers around widely used libraries (like rttr in this case):
# Wrappers/conanfile.py
from conan import ConanFile
class C(ConanFile):
    name = "Wrappers"
    def requirements(self):
        self.requires("rttr/0.9.6", transitive_headers=True)
    python_requires = "pyreq/editable"
    python_requires_extend = "pyreq.PkgBase"
  • Then there is a library that implements some standalone things using those wrappers:
# Subproject/conanfile.py
from conan import ConanFile
class C(ConanFile):
    name = "Subproject"
    requires = ("Wrappers/editable",)
    python_requires = "pyreq/editable"
    python_requires_extend = "pyreq.PkgBase"
  • Finally a more-or-less final product utilizing those things:
# BigProject/conanfile.py
from conan import ConanFile
class C(ConanFile):
    name = "BigProject"
    requires = ("Subproject/editable",)
    python_requires = "pyreq/editable"
    python_requires_extend = "pyreq.PkgBase"
  • And lastly the deduplication of Conanfile boilerplate:
# pyreq.py
import os
from conan import ConanFile, conan_version
class PkgBase(object):
    version = "editable"
    generators = ("CMakeToolchain", "CMakeDeps")
    settings = ("os", "compiler", "build_type", "arch")
    options = {
        "shared": [False, True],
    }
    default_options = {
        "shared": False,
    }

    def layout(self):
        self.folders.build = os.getcwd()
        self.folders.generators = f"{self.folders.build}/Conan/"
        self.folders.source = "."

    def init(self):
        # Conan does not inherit options by default, so docs and Slack suggest this instead
        base = self.python_requires["pyreq"].module.PkgBase
        if conan_version.major == 1:
            self.options = {**base.options, **self.options}
            self.default_options = {**base.default_options, **self.default_options}
        else:
            self.options.update(base.options, base.default_options)

class C(ConanFile):
    name = "pyreq"
    version = "editable"

It is crucial that the packages have a shared option, I could not reproduce the issue outside of our monorepo without it. I assume this relates somehow to how Conan deduces package types (shared/static/headeronly/executable) based on the options prescence or something. Here I do for x in pyreq Wrappers Subproject BigProject ; do conan editable add ./$x ; done to initialize the “workspace”, and then install the dependencies for BigProject:

mkdir BigProject/build
cd BigProject/build
conan install ..

No need to bother with CMake and sources here, as already after conan install I can see the problem. The generated BigProject/build/Conan/rttr-release-x86_64-data.cmake file has this:

$ grep 'set(rttr_' Conan/rttr-release-x86_64-data.cmake

set(rttr_PACKAGE_FOLDER_RELEASE "/tmp/c/.conan2/p/b/rttrb050f2053cedf/p")
set(rttr_INCLUDE_DIRS_RELEASE )
...
set(rttr_LIB_DIRS_RELEASE "${rttr_PACKAGE_FOLDER_RELEASE}/lib")
...
set(rttr_RTTR_Core_Lib_INCLUDE_DIRS_RELEASE )
set(rttr_RTTR_Core_Lib_LIB_DIRS_RELEASE "${rttr_PACKAGE_FOLDER_RELEASE}/lib")

, so the rttr target created by Conan for BigProject is missing the include directories. If instead of Parsing I repeat the same commands for either Subproject or Wrappers, the -data.cmake file is generated correctly:

$ grep 'set(rttr_' Conan/rttr-release-x86_64-data.cmake

set(rttr_PACKAGE_FOLDER_RELEASE "/tmp/cc/.conan2/p/b/rttr36cf223c461c8/p")
set(rttr_INCLUDE_DIRS_RELEASE "${rttr_PACKAGE_FOLDER_RELEASE}/include")
...
set(rttr_LIB_DIRS_RELEASE "${rttr_PACKAGE_FOLDER_RELEASE}/lib")
...
set(rttr_RTTR_Core_Lib_INCLUDE_DIRS_RELEASE "${rttr_PACKAGE_FOLDER_RELEASE}/include")
set(rttr_RTTR_Core_Lib_LIB_DIRS_RELEASE "${rttr_PACKAGE_FOLDER_RELEASE}/lib")

We probably do a few things weirdly/incorrectly here; I, for starters, am kinda sus about all the packages using the same self.folders.build = os.getcwd() - and therefore overwriting files of each other. I gladly accept any hints on how to improve this setup, but I have been stuck in the groundhog day of “migrate to Conan 2.0” for about a year now, and for now would very much prefer things to Just Work™ without any huge refactorings and breaking compabitility with Conan-1.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (8 by maintainers)

Most upvoted comments

Hi @Artalus

Thanks for the feedback

I got our monorepo working without the root CMakeLists.txt, but the final setup feels… weird.

There is indeed some unexpected usages there.

If you are using conan workspace install that is intended for the monolithic super-build. It shouldn’t require any further conan install at all. As shown in the docs, it is mostly:

conan new workspace
conan workspace install 
cmake --preset ...

If that is the case, the conanws.py conanfile must contain also a CMakeToolchain generator to generate a toolchain for the super-build, but it doesn’t require any intermediate conan install. Also, for the monolithic super-build, the add_subdirectory() is not recommended, but instead the FetchContent, which is more suited for the task.

And viceversa, if using a package conan install, then the conan workspace install shouldn’t be used, as that is an individual package installation, not related to the super-build, they are orthogonal.

So yes, I guess the best would be to follow up in new tickets, thanks!