depthai-core: unexpected use of CMAKE_TOOLCHAIN_FILE in cmake, recommend not using it

Please read the following in a neutral manner. I do not intend any insults. I am writing with an intention to detail my perspective and seek a solution that helps the most people. That is usually my goal. I created solutions for a billion people when I was a Microsoft employee. That experience causes me to have-no-fear, to solve hard problems, and solve problems for the most people.

Shifting the problem to someone else

My primary effort is developing apps for artists. I am the bridge between artists and complex technology. Yes, I can also dive deep into memory and threading bugs in libraries. 😉 I need my dependencies to be reliable and mange their own internals. I have no desire to dig into the internals of a dependency. It is not my job/role.

Unfortunately, I sometimes have to do the work that should be done by a dependency. I’ve done it for both OpenCV and DepthAi projects. (Remember, no blame/insults.) Going into the internals of another team is usually not a good thing. Me having to: 1) hack DepthAI’s cmake files because, 2) DepthAi and OpenCV don’t manage their own internal dependencies, because 3) OpenCV uses custom symbolic names for DLLs, and 4) OpenCV doesn’t distribute the cmake files that create these symbolic names it uses itself…ugh. No good here. 😝

All of that means I (app developer) have to review/correct the internals of OpenCV’s cmake infrastructure, write my own cmake files to match OpenCV’s internals, and hack DepthAI and/or Hunter. Only then can I do my job which is to write applications for artists. I’m smart, experiences, and I fixed all this myself. But it takes days to weeks of my time. Time is something that can never be replaced; the most valuable thing in the world. And this only fixes it for me; not the other 7.8 billion people in the world.

There is also another shift-the-problem in this discussion. I assert that shared libraries shift problems to end-users. End-users (like my artists) have problems when they don’t have the exact shared libraries needed at runtime. The shared libraries are installed/replaced/removed by system upgrades, other app installs, PATH changes, and this is well-known on Windows as “DLL Hell”. Everyone with experience has seen this. End-users usually can not fix this – they do not have the knowledge/capability.

OpenCV example

Please remember…no insults. I am not debating the pro/cons of static-vs-shared libraries. Instead, I want to examine/inspect the fears that both OpenCV and DepthAi are expressing. And I need to use static/shared libraries as it is part of this fear.

@alalek, who are you protecting? And what you protecting them from? I request you clearly document this.✍

You write, “You would get linker errors, or crashes in runtime later.” When libraries are linked together, the linker immediately reports ABI problems. That’s how a linker works. This is not a problem and nothing to fear. When a compiled/linked app is tested…errors/crashes are the goal. This is not a problem and nothing to fear. This “linker error, crashes” you write happens with both static and shared libraries. It is not unique to one or the other.

When OpenCV is compiled both as a shared-library (DLL) or as a static library it continues to produce the “linker errors, crashes” problem that alalek writes. How? Because that’s how linkers work. They report ABI problems when they link. Crashes later due to API problems are also in both shared and static – because OpenCV.DLL and OpenCV.LIB do not have all code linked within them. At runtime, both static OpenCV.LIB or shared OpenCV.DLL themselves use shared-libraries (c/c++ runtimes, all Windows functionality, ffmpeg, intel media decoders, graphics drivers, onnxruntime, tbb, ipp, cuda, opencl runtime, etc.). None of the code I listed is contained within OpenCV.LIB or OpenCV.DLL. Both can have runtime crashes.💥

The following static and shared library examples have equal chance of success or failure. I see no reason for fear.

• OpenCV.LIB built with version=13 ceres-version.lib -yet- myapp.exe is built with version=15 ceres-version.LIB
• OpenCV.DLL built with version=13 ceres-version.sharedlib -yet- myapp.exe is run with version=15 ceres-version.DLL

Distribution of static libraries is similar to distribution of shared libraries. Both are files on a hard drive. Both benefit from versioning. Both can be distributed with packages (linux rpm/deb, windows msi/exe, vcpkg, chocolately, etc.) and all those package tools have dependency management. All compilers and linkers have options to override default search paths for libraries and full path names to libraries can always be used. Both cmake find_dependency() and find_package() have versioning capability. If something specific is needed, then projects should specify it.

Is OpenCV suggesting that OpenCV has a problem when a random person on the Internet downloads OpenCV source code, compiles it, and posts a ZIP file of the OpenCV.LIB or OpenCV.DLL with no versioning, packaging, or cmake? Is this random developer the primary focus of the OpenCV team? Even if this was user#1, this random ZIP download of OpenCV.DLL or OpenCV.LIB both have 3rd party external dependency challenges and both require those dependencies be installed before a compile/link/run will succeed.

Libraries (both shared and static) provide their own ABI/API guarantees. Compiling with a different version LIB, or having a different shared library/DLL installed, are both challenges. Though, I believe the shared library/DLL is worse due to DLL Hell.🥵

My focus is on my app…not OpenCV. When I compile my app and link to static OpenCV, I am guaranteed to have that exact version of OpenCV because it is linked/contained within my app. I can test it. In contrast, if I link to shared-library OpenCV.DLL, I can test it but I can’t easily control what version of OpenCV.DLL is found on the end-user’s computer. Proof? 🔍 One of the most common complaints and problems with OpenCV on the internet is “Which version of the OpenCV DLL works with this app, 32/64-bit, os version” and “How do I install the OpenCV Dlls”. Do a quick search with Google. End-users have shared-library “OpenCV.DLL Hell”.

DepthAi example

Depthai, yesterday proved that the fear alalek writes applies to both shared and static libraries. I found a mutex bug in DepthAi where the mutex was not being locked. Correctly locking the mutex causes a chain reaction that exposes bugs in other code. The ABI and API are compatible before and after my mutex fix. However, if depthai was compiled as a shared-library, then all apps which linked to a shared DepthAi library would now randomly crash due to timing/race conditions. If DepthAi was compiled static, then the apps wouldn’t crash because they would continue to use the older code that has the mutex bug that the app’s code was lucky to not expose. You can easily reproduce this using DepthAi’s own ctest suite of tests. Before fix, all pass. After fix, 1 to 3 will usually fail due to timing/race conditions.

Depthai, you will also have OpenCV.DLL hell. Some developer will use shared-library version=4.5.6 OpenCV.dll to build depthai-core.lib. This means anyone that links to that depthai-core.lib will need a version of OpenCV.dll that is 100% compatible with version 4.5.6. Then the same dev (or another dev) will write an app that uses both OpenCV and DepthAi apis, and compile/links to version=4.0.1 OpenCV.dll to build their app. Naturally, this means the app attempts to link to both depthai-core.lib and the OpenCV.dll library stubs. This is OpenCV.DLL hell because you can’t have two different versions of OpenCV.dll linked/loaded in the same app. OpenCV might say “we are ABI/API compatible” but that is not true (because OpenCV had bugs, fixed bugs and added/changed features) and now the shared-library OpenCV.DLL exhibits the same problems alalek wrote above “linker errors, or crashes in runtime later”.

Test as soon as possible

Is ABI and API compatibility something to fear? No. All projects that matter will test for ABI/API compatibility issues. And those ABI/API compatibility issues exist for BOTH shared and static libraries.

All projects that want reliability do testing. The remaining untested random projects need automation to manage internals/dependencies. These random projects need to be able to do a simple cmake --build. They should not do the 4-step reverse-engineering and hack that I describe above; and they might not have the capability to do it.

So what is this fear of static libraries? And who is being protected?🤔

Static libraries are tested at compile/link-time when developers and testers are actively examining and testing. My testers have problems. Good! That is their job! It keeps the problem with the testers, not my end-user artists. Shared libraries should also test at compile/link-time. But they can’t test the end-user’s computer because of OpenCV.DLL Hell.

Shifting Redux

From what I can discern, this aversion to static libraries is an attempt to shift the responsibility of solving problems from one’s project to downstream projects. And in my scenario, each of you (OpenCV and DepthAi) have cascaded that to my project and the other dev that reported the above linked issue. I am confident we are not the only two in the world. OpenCV doesn’t want to do it. DepthAi doesn’t want to do it. So now I and other app devs have to do it. And we have to reverse-engineer the logic and internals of both OpenCV and DepthAI. 😞

Who are you both protecting?

• It is not me. And it is not the developer in the bug I linked above. We both want your projects to manage your own internal dependencies. So if you are trying to protect developers from an unspecific/rumor/fear of static libraries…please don’t. We don’t want that protection.
• If we developers want a specific version of a dependency like OpenCV, we will use the standard cmake methods like pointing to specific directories with OpenCV_DIR. Or we devs might load a package in our project like find_package(ceres 2.0.3 REQUIRED).
• Please remember, OpenCV and DepthAi are not the center of our apps. You are not the top-most dependency, not the only dependency, and often you are only minor includes. By not handling your own internal dependencies, you break the chain of libraries/dependencies and cascade that break down and across the dependency chain.

We devs appreciate the functionality your libraries provide. Thank you. 🙂 For some like DepthAi we directly pay money. Others like OpenCV are indirectly paid through sponsorships from companies like Intel and we buy Intel hardware. There is compensation for the effort occurring.

Unfortunately, your projects are making it harder for us developers. Giving us vague protection for something we don’t want. I know you are not doing this with malice. Both your teams have demonstrated a willingness to make things better. I suspect your projects’ fears above are based in outdated rumors rather than current fact. Is this fear based on past/outdated use of static libraries before today’s versioning, package control, and modern cmake? Or perhaps it is self-protection. Are you trying to protect your own dev/test resources so they don’t have to update your cmake code? Or, are you trying to protect your support teams from something?

Need user scenarios

I request you write out a full user scenario. This is usually 2-3 paragraphs. Describe this “person” that you are protecting. Describe two specific problem scenarios of this person. The ten-word-fear that alalek writes above is too short and vague to be actionable or follow thinking/logic.

With clear user scenarios, we as a group can probably find a solution that works for us and the most people. That is my goal. Remember, I’ve already hacked a workaround for my own project. I want to fix it for the other 7.8 billion in the world.

I agree - I think CMake does offer everything needed to correctly define all dependencies, between libraries, etc… Although I think that the pain point is distribution of static libraries (shared libraries seem to be given more thought).

That said, regarding original issue, I’ll try to find some time to play around with the toolchain issue. Any pointers on this or minimum example which isolates the issue? Otherwise will try to create a similar case locally

There is strong recommendation to use shared builds instead. Redistribution of “static builds” are over-complicated task for projects with external dependencies. Just find_dependency() approach may work until that provides the same results (you can play with find_dependency(LAPACK) on different user systems). Much worse case, this automatic approach may silently find a different version of binaries which is not compatible with used dependency headers in compiled binaries. You would get linker errors, or crashes in runtime later.

Unfortunately there is no robust easy-to-use solution for “static builds” problem. In OpenCV project we don’t add anything to “static builds”, so advanced users should configure dependencies yourself (with full responsibility) before calling find_package(OpenCV) of “static builds”. Such configurations are specific for their use case only, build environment and used options/dependencies.

@diablodale please point me to modern CMake projects / examples properly doing transitive static dependencies. I’m wondering how are they are passing down these dependencies, how well it plays with other CMake projects, etc… I’d gladly take a look as I too had a bunch of static library problems in the past (although before modern CMake came to be the norm).

As of right now, they way our library handles static dependencies is by installing them alongside. So compiling DepthAI statically (and disabling optional opencv support, as we do not handle OpenCVs dependency in same manner as our internal ones) will install all needed dependencies along side of it, making a find_package(depthai) correctly resolve everything needed to compile it into your project. This works for shared compilation as well.

I’m not sure if our way is correct, pretty sure it wouldn’t play well with installing things to system directories, but for starters that is the way I chose to do it.

I agree with @alalek on this as far as my limited insight in various different projects reach. Static dependencies are usually not handled transitively. (offtopic, in our project we do install to build directory by default and install the static dependencies that were used. This enables the option of linking to statically built depthai, and all dependencies will be resolved to ones that were installed alongside. This isn’t quite doable in case when you want to install to a public location, with public/user controllable dependencies, but okay for isolated case like ours IMO)

Having find_dependencies (find_package) in *Config seems okay, but would have to be done only for static builds. Shared have dependencies already collapsed. On one hand it seems reasonable for a project to also specify the location of its dependencies it used for itself, but given that installation main point is to redistribute, it doesn’t make much sense (maybe if those were packaged together). That issue is then similar to runtime question: every app self contained vs dependent on other libraries outside itself - but for build time.

@diablodale WRT HunterGate call, if you want you can create an MRE and I’ll take a look, but its likely better to be addressed to Hunter project instead.

About the:

That would never happen. As an almost-rule, xxxxConfig.cmake files are programmatically generated during config/compile/install of that project. These Config files live within the project installation itself. They are, in general, not hand made. A project like OpenCV is complex and I would never attempt to replicate the Config file OpenCV programmatically creates. Too complex. Too easy to make errors. And as a manpower evaluation…my one line proposed fix to the depthai cmakefiles.txt is faster, simple, and 100% works.

I meant not to recreate OpenCVs Config file but to “extend” it with dependencies. That’d mean 1 include line and the rest are the same lines that you’ve posted above:

include(path/to/OpenCVConfig.cmake)

set(OpenCV_STATIC ON) # because my opencv is a static opencv
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
    set(OpenCV_DIR "C:/repos-nobackup/opencv/.install/Debug")
else()
    set(OpenCV_DIR "C:/repos-nobackup/opencv/.install/Release")
endif()

set(Eigen3_DIR "C:/njs/vcpkg/installed/x64-windows-static-md-v142-sdk10b17134/share/eigen3")
find_package(Eigen3 3.3.7 CONFIG REQUIRED)

list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/.vscode") # contains my custom FindIPP.cmake because there is no cmake standard find module for IPP
set(IPP_STATIC ON)
set(IPP_DIR "C:/Program Files (x86)/Intel/oneAPI/ipp/latest")
find_package(IPP 2021.1.0 REQUIRED) # COMPONENTS cc i iw)

But toolchain way is probably better

haha, the plot thickens. 🧐 We might have confluence of issues. Please do review the below. I would like to reach a consensus on the problem and its cause. Afterwards, we can work on a solution. I’m including @alalek from the OpenCV project as he has direct knowledge and I’m starting to think OpenCV owns part of a solution.

My learning on this topic has been via cmake official docs, random articles found with searches, and discussions in context of projects (e.g. github issues). As I was doing a google search for suggested reading, I ran across https://cmake.org/cmake/help/git-stage/guide/importing-exporting/index.html

All required dependencies of a package must also be found in the package configuration file.

Which is the opposite of what I previously learned and wrote above in this issue. At first, my thought was…what does “found” mean? Does it mean only checking the xxx_FOUND variable? Does it mean actively searching for files on the hard drive? etc. So I did more reading at that URL and at https://cmake.org/cmake/help/latest/module/FindPackageHandleStandardArgs.html
https://cmake.org/cmake/help/git-stage/guide/using-dependencies/index.html
https://cmake.org/cmake/help/latest/command/find_package.html
https://cmake.org/cmake/help/latest/command/target_link_libraries.html
https://pabloariasal.github.io/2018/02/19/its-time-to-do-cmake-right/

Then I referenced a discussion in OpenCV https://github.com/opencv/opencv/issues/19282 and found also https://github.com/opencv/opencv/issues/18521. The first was a trigger for my viewpoint above. The latter I found today. Combining all that then made me question my previous views formed by the OpenCV issue. I can be wrong, and I prefer to learn and correct. 🔬

As a test, I traced the scenario that it is OpenCV’s job to actively search on hard drives for dependencies it needs itself. With a shared library build (e.g. Windows DLL), most everything needed is found at compile-time of OpenCV itself and everything collected and linked together in one gigantic DLL. All code needed to run OpenCV is within that one DLL.

In contrast, a static library builds of OpenCV creates a Windows .LIB file. That .LIB file contains only the code of the OpenCV project itself. It does not contain the code of its dependencies (e.g. eigen, ceres, IPP, etc.). Generally for static libraries, the linker does not combine the code of the project with the code of the project’s dependencies into one gigantic static LIB file. The linker, in downstream projects that use the static build of OpenCV, needs to link to the code for OpenCV’s dependencies. And cmake provides mechanisms for this, e.g. target_link_libraries() with transitive dependencies.

When these dependencies are known by the cmake infrastructure of OpenCV, then somehow that knowledge needs to be given to downstream projects that link to OpenCV. The built-in technology to do this is https://cmake.org/cmake/help/git-stage/guide/importing-exporting/index.html#creating-a-package-configuration-file

# OpenCVConfig.cmake file should contain...
include(CMakeFindDependencyMacro)
find_dependency(Eigen)
find_dependency(IPP)
find_dependency(ceres)
...

I searched the entire OpenCV source code from the 4.5 branch. There is no code that calls find_dependency(). I also searched an install of OpenCV. It also does not have any call to find_dependency(). Referencing the official cmake docs above…suggests to me that it is OpenCV’s job to find all its dependencies in its OpenCVConfig.cmake file. I don’t think OpenCV is doing what it needs to be doing. This aligns with the issue https://github.com/opencv/opencv/issues/18521 and underlies issue https://github.com/opencv/opencv/issues/19282

Perhaps then a confluence of issues contribute. Please do check each of these for correctness.

1. There is no cmake config file or cmake find module provided by the IPP (Intel Performance Primitives) team. 🤦‍♀️
2. There is no cmake find module for IPP written by Kitware and shipped with cmake
3. There is no cmake find module for IPP written by OpenCV and used internally within its own project
4. OpenCV does not find its own dependencies in its OpenCVConfig.cmake as per official cmake instructions
5. Because OpenCV does not follow cmake instructions, this cascades the burden to find all upstream dependencies of OpenCV to all downstream projects using OpenCV
6. Somewhat circular, now downstream projects (like depthai) have the challenge to write their own custom IPP find module that is compatible with OpenCV (it uses very specific symbolic dependency names), and write their own toolchains to find Eigen, ceres, glog, cuda, and all other OpenCV upstream dependencies.

There may also be issues within depthai project or hunter, but these are later problems that might be sideaffects of the above six.

OK. Enough for today. What are all your viewpoints? 🙃