runtime: runtime depends on extremely recent verisons of cmake, blocking adoption on Linux

Based on the feedback here we will change our supported Cmake version from Linux to 3.6. @jkoritzinsky has started this work with #39044. In order to avoid regressing the build using older version of Cmake we will add in a CI job which specifically builds with 3.6 on CentOS.

For non-source build builds we will continue to use higher versions of CMake in the same way we use clang9 now.

/cc @dotnet/runtime-infrastructure @jkoritzinsky @omajid @jkotas @janvorli

Please let us know if there is anything else required.

This is a much more minor concern than the one cited, but this also makes it harder to get started contributing to dotnet/runtime.

For building on RHEL 7 (and CentOS 7), we have a version 3.6.2 available. This is not the default cmake, but separate build maintained by the llvm team.

That would be our desired minimum version, and support all distros mentioned in the first comment.

It doesn’t sound too crazy to me to do this as part of source-build rather than tying us to ancient versions of cmake in dotnet/runtime.

Source-build and source-build bootstrapping is already quite complex and costly adding cmake bootstrap adds to that complexity significantly. That additional cost would be a significant detractor for maintainers wanting to bring in .NET into their distro.

We also are in the midst of a major effort to make source-build more maintainable https://github.com/dotnet/source-build/issues/1510 This effort will go a long way in surfacing difficult issues like this one earlier. Indeed, that source-build effort makes any additional effort to bootstrap cmake prohibitive. cc @dseefeld

We should be able to preserve most of the new design we have if we use functions to add the right defines. It’ll just be tedious to convert all of the add_compile_definitions and add_link_options to target_compile_definitions and target_link_libraries that are attached to specific targets.

Thank you @omajid and @RheaAyase for providing all the details. It seems to me that it is inevitable to move back to using an older cmake.

Has something changed since the conversation we had in dotnet/dotnet-buildtools-prereqs-docker#185 about using a newer CMake version?

It’s my mistake I didn’t look into all the default versions back then.

I thought the conversation was essentially about RHEL 7/CentOS 7 going from 3.14.x to 3.15.x.

3.1.5.x didn’t seem any worse than 3.14.x in terms of distribution inclusion. RHEL 7, as you can see from the matrix above, only has 2.x I figured that the version of 3.x wouldn’t matter.

I didn’t realize that the matrix was affecting RHEL 8 as well as many, many, many other distributions as well.

I would like to provide some additional context that might help clarify things.

The ultimate goal of the source-build effort is to package .NET into as many Linux distributions as possible. If/when we are successful, .NET will be available out of the box in those distributions, just like may other languages like C, Go, Java, Perl, Python, and Ruby.

To be included in distributions, generally packages are only allowed to use what’s already in the distribution. If a distribution doesn’t have cmake at all, for example, there’s no way we could package .NET in that distribution. We would first have to add cmake and then try and package .NET in that distribution.

The list of distributions I described above all already have an older version of cmake. So we can’t fix the problem by adding cmake. We would have to upgrade cmake. That has an unknown (at least to me) compatibility impact. I expect that upgrading the version of cmake in, for example, RHEL 8 or Debian 9 is going to be quite a challenge. We would have to show that all programs that worked with cmake 3.x.y continue to build the same with cmake 3.14.5 (or newer). There’s a very high bar for making what could be a breaking change to one of the fundamental build tools being used in a distribution. And we have to meet this bar for all the distributions we want to add .NET to.

There’s also the question of timing. Even if an upgrade was accepted (I dont think it will be in many such enterprise or stable distributions), a change can take a long, long time. If it takes 1 year from now for Debian 9 to get the new cmake version in the main package repositories, it will be too late for use in .NET 5.

Perhaps you might understand the impact of this issue more broadly if you mentally replaced cmake with glibc in this discussion. We don’t ask users to install a separate glibc. We don’t ask distributions to upgrade their version of glibc. We look at our support matrix and try and support all the glibc versions available there.

@wfurt said:

I’m using cmake’s packages on Ubuntu without any problems. I think that covers Debian as well.

I hope it now makes sense why it doesn’t even work for the versions of Debian and Ubuntu I listed in my initial comment above. Those versions of Debian and Ubuntu do not have the required version of cmake in the main package repositories. Using an external package repository is not acceptable for Debian.

I’m wondering if it would make more sense to push forward and provide newer cmake/cmake3 on those distributions.

There’s a very high compatibility bar that needs to be crossed. And it needs to be met for every distribution that we want to package .NET in. I know .NET cares mostly about run-time compatibility. But Linux distributions generally care about as much for build-time compatibility. Updating cmake is far from a trivial operation.

It seems like we will need to spent non trivial effort to go back.

Agreed. I can help with this.

@wfurt said:

You need to add new repository. https://apt.kitware.com (probably can skip first step) With that, everything should work fine with package manager.

@jkoritzinsky said:

Kitware provides an APT feed with the most recent version of CMake for Ubuntu at https://apt.kitware.com. We link to it in the getting started docs.

That’s true, but it’s also non-usable in a distribution context. Most distributions (certainly the case for enterprise distributions like RHEL) wont break the trust model by using binaries compiled by someone else.

For the older distributions, would it be possible to build the newer version of cmake from source based on tools available in the distro and use that to build dotnet/runtime?

A general upgrade for every distribution might be next to impossible. Doing this as part of .NET Core build might something we could do as a last resort. That might still violate some distribution policies. For example last time I tried to use a new version of cmake in RHEL, I had to bootstrap it using a prebuilt cmake binary. This would not work for Fedora (but Fedora is not in the list of distributions that has and old version of cmake).

@wfurt said:

While I hate to upgrade frequently, I think packages from Kitware should be sufficient for developers and contributors.

That’s true for developers and contributors, but not builders and Linux distributions. We can’t use binaries produced by a third-party (except in very specific circumstances, like bootstrapping).

I could make it work without much troubles and I really don’t care about the source. .NET is distributed in similar way.

Community linux distributions that focus on Openness and Freedom want to be able to build everything from source and only ship things they have themselves built from source. That lets them be sure those users who do care about the source are satisifed.

Enterprise linux distributions (as well as various *BSDs, I think) care about sources because it lets them avoid risk; if an open source project can’t fix an major issue, then they can step in and fix things themselves.

Both models mean that being able to build everything from source is important. That’s why there’s a source-build project in the first place.

If this is really about official build, can this be solved with source-build @tmds?

I am trying to think what a solution that lives only on source-build-side would look like. Source-build will have to, some how, support older versions of cmake and build runtime based on that.

One way to do that would be to build a newer cmake version as part of source-build. I touched on this earlier in my reply: there may be issues with this.

Another way source-build could support older versions of cmake would be to carry patches locally to remove the dependency on newer cmake version. That would be, effectively, doing the same work that everyone wants to avoid here. Worse, it would be specific to source-build. It would be likely that some build bugs and issues will exist only in source-build. I would much rather do the changes here, in runtime, where everyone can collaborate and share issues and fixes.