conan: Unable to resolve conflicts between requires and build_requires graphs

Conan v1.13 introduced reporting conflicts between requires and build_requires graphs. This is very good, as it solves many problems, but sometimes you need to be able to resolve reported conflicts and this seems not being possible at the moment (using conan v1.13.1).

Here is a simple project that demonstrates the issue.

Basically, we have a package Aaa with three versions: v1.0.0, v1.1.0 and v1.2.0. Package Bbb requires Aaa/1.0.0, package Ccc requires both Aaa/1.1.0 and Bbb/1.1.0 (thus overriding Bbb's dependency on Aaa and package Ddd build_requires both Aaa/1.2.0, Bbb/1.0.0 and Ccc/1.0.0. However, instead of overriding both Bbb's and Ccc's requirements on Aaa with version 1.2.0, conan reports a conflict:

ERROR: Conflict in Bbb/1.0.0@user/testing
    Requirement Aaa/1.0.0@user/testing conflicts with already defined Aaa/1.2.0@user/testing
    To change it, override it in your base requirements

However, if I change Ddd's build_requires expression from

    build_requires = (
        'Aaa/1.2.0@user/testing',
        'Bbb/1.0.0@user/testing',
        'Ccc/1.0.0@user/testing',
    )

    build_requires = (
        'Bbb/1.0.0@user/testing',
        'Ccc/1.0.0@user/testing',
        'Aaa/1.2.0@user/testing',
    )

I get different conflict:

ERROR: Conflict in Ccc/1.0.0@user/testing
    Requirement Aaa/1.1.0@user/testing conflicts with already defined Aaa/1.0.0@user/testing
    To change it, override it in your base requirements

Also, other permutations of build_requires order also give different conflicts.

My questions are:

why does conan report conflict here? Why it doesn’t override Bbb's and Ccc's dependency to Aaa/1.2.0?
how can I solve this conflict? Build_requires expression does not support override keyword (a solution I would use if Ddd required Aaa, Bbb and Ccc instead of only build_required it).
if I replace build_requires with requires in Ddd's conanfile, then no conflict is reported, but I don’t want Ddd to carry Aaa, Bbb and Ccc as dependencies to downstream and I don’t want to them to affect Ddd's package_id. (Image that Ddd is a package that contains only executable built against Aaa, Bbb and Ccc static libraries and consumers of the Ddd package do not need executable rebuilt if they themselves use different versions of those static libraries (you can think of them as utility libraries)).
why does the order of packages in build_requires matter? Why can’t I simply sort all my dependencies alphabetically? (we noticed the similar problem also for requires clause, although in more complex examples).

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 1
Comments: 21 (18 by maintainers)

Commits related to this issue

Fix misuse of build_requirements for Conan 1.13+ Looks like we were misusing build_requirements See https://github.com/conan-io/conan/issues/4753#issuecomment-481663245 and https://github.com/conan-... — committed to thorntonryan/conan-requirements-overriding-change-in-behavior by thorntonryan 5 years ago

Most upvoted comments

Ok, there is too much feedback, lets start with the case of @DoDoENT

I have put the zip in a repo here: https://github.com/memsharded/tools_build_requires, with a build.py script to run everything. I have added a clear output of the all_keys option to make sure the correct value is used in the different contexts.

Lets start with this part of the graph:

Some pre-conditions:

The OCRModelBuilder EXE can only link with one version and configuration of all the other static libraries.
If at some point the dependency graph of OCRModelBuilder has two different versions or two different configurations (like all_keys=True and all_keys=False) of the same dependency, that is a conflict, and an error should be raised, and the build should not continue as the final result would be buggy, as it is not determined which package version and configuration will be linked

The expansion of the graph IF OcrModelBuilder uses build_requires as currently defined would be something like:

This graph had 2 important bugs in conan 1.12 and older versions:

There are several versions of CoreUtils that try to be linked in the Exe. Which version was finally being used was not deterministic and totally quiet, the different versions of CoreUtils were dumping their information in deps_cpp_info, and only the last version that did it was the one finally used.
There can be different configurations of the Protection package doing the same overwrite. If the consumer package OcrModelBuilder didn’t explicitly change the all_keys, there would be 2 different configurations, and the one finally being used was also not deterministic.

With the new expansion model defined in Conan 1.13, those bugs have been fixed. Conflicts will be raised, and the final dependency graph when the conflicts are solved, will resemble the first figure, with diamonds. So far, this behavior is not a considered a regression and it won’t be reverted, the conflicts totally make sense and are the evidence of something that in the past was failing silently. I am not saying that in some cases the final result couldn’t be a good one, but that was mostly a lucky coincidence.

Now the problem is that build_requires do not behave like normal requires. They do not override the same way, because they are essentially private to the package that declares them. If you want to make them work correctly, you need to correctly align the versions used in the different packages requires and build requires. I think the underlying cause of this discussion is the abuse of build_requires trying to approach a different use case that the one they were designed for. Please let me summarize some critical points from the docs:

There are requirements that are only needed when you need to build a package from sources, but if the binary package already exists, you don’t want to install or retrieve them.

These could be dev tools, compilers, build systems, code analyzers, testing libraries, etc.

They can be very orthogonal to the creation of the package. It doesn’t matter whether you build zlib with CMake 3.4, 3.5 or 3.6. As long as the CMakeLists.txt is compatible, it will produce the same final package.

Build requirements do not affect the binary package ID. If using a different build requirement produces a different binary, you should consider adding an option or a setting to model that (if not already modeled).

That means, they are designed for tools like cmake or gtest. Not for libraries to be linked with. They shouldn’t affect at all to your binary. It doesn’t matter if you are using the CMake from your system, or from a Conan package, and it doesn’t matter if you are building your tests linked with gtest or not, the final package (headers, library), should be the same.

This is definitely not the case for an executable linking with libraries. They are really required, they affect the binary, they are not tools that will be installed in the system. Another view, would be what happens if now the libraries are shared instead of static. Then, a build_require will not work, as the dependencies are necessary not only at build time, but also at consume time. That could be addressed with some conditional logic that uses build_requires if the some option is shared=False, and regular requires if the option is shared=True, but that construct would be a clear smell.

So, the first stopper, which is building OcrModelBuilder is solved if we use regular requires instead of build_requires in it. Now the problem is moved downstream when we want to use this OcrModelBuilder as a build_require for EuropeanOcrModels, which is trying to do:

The different versions of CoreUtils and the different configurations of Protection (with and without all_keys) will conflict, because EuropeanOcrModels need to build, it will build_require OcrModelBuilder, and that will transitively try to plug the incompatible dependencies.

So, what could be a possible approach right now, with the current provided tools? I think you already realized that 2 levels of build_requires is what it is really needed to fully isolate dependencies when they are joined at build time in the same package, coming both from the regular requires and from build_requires. So, as all problems in SW Engineering, I’d say this could be approached adding a level of indirection:

The code is in the github repo: https://github.com/memsharded/tools_build_requires.
It defines a OcrModelBuilderTool that repackages the OcrModelBuilder, according to: https://docs.conan.io/en/latest/devtools/running_packages.html#runtime-packages-and-re-packaging
The OcrModelBuilderTool could probably define settings = "os_build", "arch_build" for correct cross-building, and isolating the build settings (a single OcrModelBuilderTool executable could be valid for different release/debug and compiler versions settings)

The graph would be:

This approach has some other nice properties:

The OcrModelBuilder would contain only the executable.
The OcrModelBuilderTool package will repackage the executable, but it might be able for example to repackage the shared libraries, if the option shared=True was used to build the OcrModelBuilder.
The result would be an independent OcrModelBuilderTool that can be used in the same way a CMake package would be use.

Please check the repo at https://github.com/memsharded/tools_build_requires, it implements exactly this later graph

So, in my opinion, the changes introduced in Conan 1.13 and maintained in Conan 1.14, (modulo the bug fix pending for Conan 1.14.2 in https://github.com/conan-io/conan/pull/4937) are valid, and actually protecting from very dangerous effects of abuse of build-requires, like linking the wrong dependency version, or what could be worse, to link with the wrong all_keys=True option in production without even noticing.

Now, it is true that a mechanism to be able to depend on tools with build_requires, when those tools also have transitive and incompatible dependencies with our main dependency graph is necessary. I think the possible approach with current tools is the creation of a dedicated “Tool” re-package. Also, this should probably be the point where a new model is necessary, to specify that EuropeanOcrModels depends as build_require to OcrModelBuilder, AND that it doesn’t need its transitive dependencies. Or maybe a characteristic in OcrModelBuilder, that marks the package as a “Tool”, and it knows how to model that regarding the transitivity. Those are very preliminary ideas, and it would be a complicated thing, but I feel that the real solution goes in this direction.

Looking forward your feedback, if necessary we could even do a call/chat in slack or zoom.

Thanks very much for all your detailed feedback, your prepared code, etc., you rock! It is great to be able to work this way 😃 😃 😃

memsharded on Apr 10, 2019

With the official release of Visual Studio 2019 this now begins to pose a serious problem. Conan 1.12.3 does not work correctly with cmake generator for VS 2019 (it sets generator to “Visual Studio UnknownVersion 16 Win64”) and latest conan version (which handles VS 2019 correctly) does not support resolving conflicts between build requirements.

We really need to get this fixed.

DoDoENT on Apr 3, 2019

Wow!! That’s really huge feedback! Thank you very very much! I will definitely try this approach with another level of indirection (i.e. the OcrModelBuilderTool package) and report back if I stumble upon new problems.

However, since I am currently in the middle of another project, it will probably take me a few days to find time to try this, so stay tuned!

Anyway, @memsharded, this comment of yours is so detailed and great that it could be reused as a case study of how Conan can be used in rather complex development environments (maybe a blog post? or special part of documentation with case studies?). Of course, I first need to check that it really works for my real use case.

DoDoENT on Apr 10, 2019

This issue was reported in 2019 and it seems, it will be long time “known issue”. I think, it is time to document it.

https://docs.conan.io/en/latest/versioning/introduction.html ‘Version and configuration conflicts’ section should include information about this limitation

AndreyMlashkin on Apr 19, 2022

Hi @kad-beekw

When you do install, make sure to use both --profile:host myprofilehost --profile:build myprofilebuild. That will allow to coexist 2 openssl, one in the build context, dependency of cmake, and the other in the host context, dependency of qt.

This will be the mode of operation in Conan 2.0.

memsharded on May 13, 2021