conan: "Case insensitive filesystem can't manage this"

As @memsharded already indicated, there are basically two different solutions. I’ll try to formulate them from a user’s point of view:

Alternative A

Conan recipe names are case-sensitive, i.e. openssl/1.1.1d and OpenSSL/1.1.1d may coexist and are considered to be independent from each other. This is is the status quo.

This means, that the Conan cache implementation on Windows is broken, as it cannot store both recipes at the same time.

Possible fixes:

• Generate random or hashed directory names for the cache and maintain an index file which maps package names to directories.
• Implement an encoding which maps arbitrary package names to lowercase-only names and vice versa. E.g. OpenSSL <-> $open$s$s$l ($c stands for an uppercase letter C; $ is forbidden in conan recipe names).

Alternative B

Make Conan recipe names case-insensitive. I.e. openssl/1.1.1d and OpenSSL/1.1.1d will represent the same package. This is a breaking change. As various comments in this and related issues stated, this raises questions on how to display the name to the user and how to handle search queries.

Possible implementation: Convert names to lowercase. Adapt dependency resolution accordingly. Uppercase directories in cache are either migrated or ignored.

Summary

I think this issue is quite critical, since conan-center has transformed all recipe names to lowercase recently. On Windows, a Conan cache cannot be shared anymore between two projects where one uses an old version of a recipe from conan-center (mixed case) and the other uses a recent version (lowercase). This is a huge problem in CI. Therefore many projects are stuck with old versions from conan-center until this issue is fixed. In my opinion, alternative A is the way to go since only the cache implementation must be changed and - most importantly - it’s a backwards-compatible change and easy to understand. I also think that any approach, that tries to block uploads/exports (like suggested here) or errors on consumer side is no real solution in a distributed system.

If companies cannot convert their packages to lowercase for Conan 2.0, how will they upgrade to Conan 2.0 …?

They won’t unless there will be a possibility for co-existence of Conan v1.x and Conan v2.x projects. No company will migrate all their packages to lowercase and to other conan v2.0 features all at once. The transition will need to happen gradually, one project at a time, during a certain period. This requires a period where single conan cache has the support for both old and new packages (i.e. OpenCV and opencv, etc.).

Conan 2.0 is a great opportunity for cleaning the house, getting rid of technical debt and bad defaults.

I agree with that. But keep in mind that lot’s of houses are “too big” to be cleaned at once. You need to start with one room, then move on to the next and so on… And during that period you will be in a state where some rooms are cleaned and others are not.

But if full backwards compatibility is required, then there is little we could do, let’s continue doing incremental Conan 1.X releases, and lets keep adding debt and slowing development. If changing the package names to lowercase is “unfeasible”, then is it really worth kicking off Conan 2.0, that will require bigger changes than that?

I wouldn’t say that full backwards compatibility is required. Such a thing would definitely make Conan v2.0 pointless. However, co-existance of v1.x and v2.0 packages for different projects on the same machine is required. Without this, the transition to v2.0 would be “all or nothing”, and most companies will choose the “nothing” option. Just remember what happened with pyhon 2 -> python3 upgrade - more than 10 years have passed and some projects are still locked to python 2. It would be shame to let happen the same to Conan.

Let me clarify - I don’t expect for conan v1.x project to be able to consume conan v2.0 packages or vice versa. I’m quite OK with breaking this. But, in order to migrate 200+ packages from conan v1 to conan v2 I’ll have to start one package at the time, updating dependencies. Thus, the “incompatibility border line” will spread from a single package at first to the majority of the packages in the future. But during that, the developers on the most downstream packages should be able to work on them and make product releases, while having a separate development branch on their machine where they could work on migration to conan v2.0 and using v2.x-compatible dependencies. This means that, if I today rename my internal OpenCV package to opencv, it will take approximately 2 months to propagate it to the most downstream users that are building final products. In those 2 months, our CI servers and machines used by senior developers and project managers that work on both low-level (“most upstream”) and high-level (“most downstream”) projects will need to handle caches with both conan v1 and conan v2 packages.

Of course, one possibility would be to maintain two local caches on each machine - but this looks to me like tool’s job, not a developer’s job (maybe conan v2.0 can use different default locations for cache than v1.0? That would immediately make v1 packages “invisible” to v2 projects and vice-versa).

This decision, as many times, is a trade-off. I believe that going all lowercase will indeed be very beneficial for users, despite the added migration inconvenience.

I agree. But we need to find a trade-off to minimize the migration inconvenience, not create another pyhon2 -> python3 fiasco.

Just my opinion, will probably ask for feedback in the Conan 2.0 Tribe.

About that, has the mailing started? I didn’t get any notification about any message.

I have just tried a conan install for that dependency, with --build=missing in Windows 10, VS14, and it worked.

Is it possible that you had tried another package recipe, that was still in your local cache? It might happen that you installed some opencv/xxx@user/channel, and that difference in case will cause trouble in Windows, that is why the system might be warning.

Did it fail at the very end? Can you please copy the output lines before that? What is the output of conan search?

Try clearing your cache with conan remove "opencv*", and repeat.

Yeah, when Conan 2.0 is released and stable, I’m sure we will address that during if we decide to migrate. But for now we’re using Conan 1.x

We expect 2.0, hopefully, before EOY. Also, all necessary recipe upgrades have been backported and can already be implemented in 1.X, and that is the recommended approach anyway in 1.X, because the legacy tools will not longer be maintained. Changing the package names to lowercase can also be done (and should be done) in 1.X, not in 2.0, because the idea is that updated recipes will be compatible 1.X<->2.0.

But that will be another, bigger problem. If companies cannot convert their packages to lowercase for Conan 2.0, how will they upgrade to Conan 2.0, that might require bigger changes in their recipes (to use toolchain() approach, changes in the graph model, deprecation of exports_sources in favor of the export_source() method, complete removal of some generators, etc, etc)?

Conan 2.0 is a great opportunity for cleaning the house, getting rid of technical debt and bad defaults. But if full backwards compatibility is required, then there is little we could do, let’s continue doing incremental Conan 1.X releases, and lets keep adding debt and slowing development. If changing the package names to lowercase is “unfeasible”, then is it really worth kicking off Conan 2.0, that will require bigger changes than that?

To summarize:

• Making OpenCV and opencv 2 different, unrelated packages, was a bad default, confusing, unsafe and problematic.
• Going “case insensitive” is definitely the way to go in Conan 2.0
• True, full case insensitivity will add a bunch of complexity to the codebase, will have challenges in Windows and OSX and be more error prone for the program itself, but also confusing for users, because they will be able to write boost/1.64, Boost/1.64, BOOST/1.64 to refer to exactly the same package. 2 different servers might contain the “same” package, but with different casing. You could install boost/1.64 and get BOOST/1.64 in your cache paths, or get Boost/1.64 in the same cache path, installing from different origins.
• Forcing all packages being lowercase will be extremely simple in the codebase, robust in clients and servers, immutable everywhere, easy validation. The only disadvantage is that it requires users to convert to lowercase. As there will be necessary changes anyway, it seems a very reasonable tradeoff. And Conan 2.0 is starting this month, there will be many months until it is released, there is plenty of time for such migration.

This decision, as many times, is a trade-off. I believe that going all lowercase will indeed be very beneficial for users, despite the added migration inconvenience. Just my opinion, will probably ask for feedback in the Conan 2.0 Tribe.

Well, yes, it is possible to introduce changes that could be considered non-breaking if we use configuration opt-ins, and then tell that as you opted-in, you agreed that something was changing.

So, in my opinion, this is massively underestimating the development and maintenance effort of this feature. There are tons of things that can and will go wrong, overwhelming (even more, if that is possible) the maintainers in support, bug fixing, releasing new patches versions…

Just to cite a few things that come to my mind:

• Build systems will fail when finding unexpected characters like $ in the path.
• Templates, user file manipulation or scripts will fail when finding $ in the path.
• Packages in windows that are named “THIS_PACKAGE_NAME_IS_VERY_LONG”, will largely increase its path length, with increased likelihood of hitting the Windows path length limit and failing.
• User scripts that access the cache will fail, and will re-fail if for some reason we need to change the encoding of the capital letters
• Conan version downgrades will crash badly.

Hyrum’s law, all of these scenarios will inevitably happen. This is not counting the regressions and bugs that will be introduced while coding this feature.

But another important problem with this huge effort is that it keeps perpetuating something that is not a safe behavior:

• Making different casing being different packages, like opencv and Opencv a different package, is a potential security risk known as “typo-squatting”. As the Conan ecosystem thrives, and more remote servers might be used, it will eventually become yet another attack vector.
• The cognitive overhead and confusion for users, for example when they inspect the Conan cache. Let’s say that in Conan 2.0, the defaults change and the opt-ins are enabled and hardcoded by default. Now users when they go to the conan cache, they will start finding $ in the paths there, that they need to remember when they are “tab-autocomplete” their commands. Using one certain casing for one package doesn’t allow to override or conflict avoid from the consumers. Different casing will keep increasing the chances of conflict. The provides feature can help to raise conflicts, but not to solve them.

So the current proposal for Conan future is to make package names case insensitive, not to allow and manage different casing as different packages. And the most straightforward, robust and understandable way for the users, is to force all package names to be lowercase always. At the moment this is the only possibility I see that will keep the sanity of both users and maintainers.

Thanks @Aalmann for the analysis. It is completely correct.

We will be working on this to be fixed, the alternatives would be:

• Internal lower case handling. I like it, it is probably the simplest to implement and the easiest to maintain with robust behavior. The only possible dissadvantage is that losing the user cased package name might be a bit confusing/annoying to users strongly relying on CamelCase for their package names. Conan search results, conan output, etc, will always output lowercase named packages.

• Internal case-insensitive, storage, comparison and hashing. This is way more complex to implement and maintain and might be more difficult to achieve robustness. Commands like conan search or the conan output will output the original user-defined casing. It has the problem also of consistency, you can also have internally different user-defined casing for the same package, it is not fully clear to me how those should be handled, always respect the first one? What happens if you have different cased names from different Artifactory repositories?

I am not sure if you are suggesting in your conclussion the first one or the second one, as in your search output you actually output upper cased Boost package.

Having a similar issue with Boost:

>conan download boost/1.69.0@conan/stable -r conan-center

Downloading conanmanifest.txt
[==================================================] 125B/125B
Downloading conanfile.py
[==================================================] 31.6KB/31.6KB
Downloading conan_export.tgz
[==================================================] 602B/602B
Decompressing conan_export.tgz: 610B [00:00, 13.0kB/s]
ERROR: Requested 'boost/1.69.0@conan/stable' but found case incompatible 'Boost'
Case insensitive filesystem can't manage this

When using uppercase I get this:

>conan download Boost/1.69.0@conan/stable -r conan-center
ERROR: 'Boost/1.69.0@conan/stable' not found in remote 'conan-center'

I’m on the following system:

• Windows 7 Prof 64bit
• Python 2.7 (yes, I know…)
• Conan 1.11.0 installed via pip

As this seems to be a package issue I’ll reference this to the corresponding package as well. Just added this here because it seems to be a reoccuring issue that perhaps needs to be solved (or at least handled) within conan directly. Perhaps it’s possible to notify the package maintainer before uploading that cases are mixed…

Update:

• I do not have any Boost* packages in my local conan cache
• I saw that the package name in conanfile.py is written in lowercase boost
• The package on conan-center is also written in lowercase boost -> Not sure why the system thinks that uppercase Boost is found…

Update2: Ok, understood a bit better what’s going on. I had a directory Boost in my local conan cache, but tried installing a package which was named boost. Of course this causes issues on Windows when things have the same name all of a sudden. Not sure how to prevent this from Conan, it’s not even an issue with the package itself, but with the user using multiple packages spelled differently. Long story short: Removing the Boost directory from my local conan cache solved the issue.