bioconda-recipes: openssl 1.1.1 -> openssl 3 change in conda-forge causing broken installations?

many packages on bioconda, e.g. htslib, depend on openssl 1.1.1
conda-forge recently removed openssl 1.1.1 from their build matrix:
- https://github.com/conda-forge/conda-forge-pinning-feedstock/commit/4c2a87622e6f9ecdac3f983892d736d58ca10e4d
this means new conda-forge packages are only built against openssl 3

I think this is causing some strange issues for me with dependency resolution:

If I do mamba install "python=3.9" pbiotools, the packages to be installed include

  + openssl                    3.0.8  h0b41bf4_0           conda-forge/linux-64     Cached
  + pysam                      0.9.1  py39h838ef2a_12      bioconda/linux-64        Cached
  + python                    3.9.16  h2782a2a_0_cpython   conda-forge/linux-64     Cached

Apparently this old pysam package is considered compatible with openssl 3 when dependencies are resolved (I think because it depends on htslib 1.3.1 1 which doesn’t include openssl as a dependency), but this installation is broken:

Python 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lkeegan/mambaforge/envs/ssl_versions/lib/python3.9/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
ImportError: libcrypto.so.1.1: cannot open shared object file: No such file or directory

If I also pin openssl, mamba install "python=3.9" pbiotools "openssl=1.1.1", then I get a working install with a much more recent version of pysam:

  + openssl                   1.1.1t  h0b41bf4_0           conda-forge/linux-64     Cached
  + python                    3.9.15  h47a2c10_0_cpython   conda-forge/linux-64     Cached
  + pysam                     0.20.0  py39h9abd093_0       bioconda/linux-64        Cached

Would it make sense to temporarily add openssl ==1.1.1 as a dependency to the pbiotools package as a workaround for this issue?

And are there any plans to follow conda-forge and migrate to using openssl 3 on bioconda?

Thanks!

About this issue

Original URL
State: open
Created a year ago
Reactions: 1
Comments: 28 (28 by maintainers)

Commits related to this issue

Add rp-bp - https://github.com/dieterich-lab/rp-bp - https://rp-bp.readthedocs.io - require pysam>0.9.1 to avoid #39356 — committed to lkeegan/bioconda-recipes by lkeegan a year ago
Update pbiotools - require pysam>0.9.1 to avoid #39356 — committed to lkeegan/bioconda-recipes by lkeegan a year ago
Add rp-bp (#39210) - https://github.com/dieterich-lab/rp-bp - https://rp-bp.readthedocs.io - require pysam>0.9.1 to avoid #39356 — committed to bioconda/bioconda-recipes by lkeegan a year ago
Update pbiotools (#39374) - require pysam>0.9.1 to avoid #39356 — committed to bioconda/bioconda-recipes by lkeegan a year ago
Add rp-bp (#39210) - https://github.com/dieterich-lab/rp-bp - https://rp-bp.readthedocs.io - require pysam>0.9.1 to avoid #39356 — committed to cokelaer/bioconda-recipes by lkeegan a year ago
Update pbiotools (#39374) - require pysam>0.9.1 to avoid #39356 — committed to cokelaer/bioconda-recipes by lkeegan a year ago

Most upvoted comments

So, here is the PR which fixes all old unpinned packages: https://github.com/bioconda/bioconda-recipes/pull/39812

johanneskoester on Mar 9, 2023

Correct, the repodata patching will only prevent old packages from incorrectly being used in the environment solution, there’s nothing that can make existing packages compatible aside from recompiling them. We’ll need a migration for that, which I guess we need to do sooner rather than later.

dpryan79 on Mar 4, 2023

@jmarshall I completely agree. To me, the broken envs that are generated now are the bigger problem, that’s why I want to solve it first. But yes, afterwards, we should definitely try to get to the update.

johanneskoester on Mar 9, 2023

There are two categories of problem:

The (very) old packages that omit openssl version constraints, which @johanneskoester focuses on, and which can be fixed by repodata patching.
The lack of any viably up-to-date e.g. htslib packages compatible with recent conda-forge packages, which is why the OP opened this issue.

Hence:

Answer 1: The existence of htslib=1.17+openssl=1.1 packages is not a problem. But the non-existence of htslib=1.17+openssl=3 packages is a problem, has been a problem since late January, and will become a bigger problem as time goes on.

jmarshall on Mar 9, 2023

I came to this thread late. I have some questions.

I didn’t recognize that openssl=3 was an issue but did know that htslib=1.17 needed updating. So I used the recent htslib pinning change in builds on the bulk branch. The result is that we now have htslib=1.17 but it is using openssl=1.1.

There were 159 total packages that had htslib-induced updates from bioconda-utils update-pinning. Some of those are not building on bulk and need to be fixed.

mamba repoquery whoneeds htslib=1.17 -c conda-forge -c bioconda says there are currently ~75 recently-built packages that depend on htslib=1.17 which was built with openssl=1.1.

Note that with these new builds, the original reason @lkeegan opened this issue for appears to be solved.

Question 1: Are these htslib=1.17+openssl=1.1 packages a problem?

Question 2: What is the best course of action moving forward?

I think it would be the following:

Fix the remaining packages failing in bulk that were recently bumped due to depending on htslib. Then at least we will have a complete “set” of packages that build against openssl=1.1 + htslib=1.17.
(finally) use the updated conda-forge-pinning. This will be a lot of rebuilt packages. But over the next week at least I should have bandwidth to babysit the bulk builds. @jdblischak @jmarshall it would be great to get help fixing broken packages during this time (and anyone else you can gather…)
Develop a plan for more frequent pinning updates. This has been under discussion for a while; fundamentally it is a resources question (human resources and compute resources). No good answer here.

Question 3: How to select which conda-forge-pinning to use? Naively I would just grab the latest available. But I know @mbargull had concerns regarding pinning the cxx compiler, and I don’t know how to figure out if jumping over 6 months of pinning updates is otherwise problematic.

daler on Mar 9, 2023

The openssl depedency is constrained to “openssl >=1.1.1q,<1.1.2a”, because it was built against openssl 1.1.1. We could patch this dependency to the more relaxed “openssl >=1.1.1q,<4.0a0”,. This would enable the conda solver to install recent versions of htslib in the same conda env as openssl 3 and the most recently updated conda-forge binaries. But this seems dangerous.

This would not work at all, because the soversions are different: openssl 1.1.x provides lib/libcrypto.so.1.1 while 3.0.x provides lib/libcrypto.so.3. So the parts of the htslib package that are linked against libcrypto will fail to execute at all, because the shared library they are looking for is not present.

(This category of problems occurs largely because conda does not track soversion dependencies.)

However because bioconda’s htslib package has been built with plugins enabled, if you were to lie about the constraints as suggested then htslib and programs that use it would in fact likely work fine. Only S3 file access uses libcrypto (i.e., only libexec/htslib/hfile_s3.so is linked against libcrypto, and it is dynamically loaded by htslib), so only attempts to access S3 files would fail.

So that’s perhaps not so bad. However the real answer is to migrate to building against openssl 3 instead of / (temporarily) alongside openssl 1.1, which is indeed now something that is overdue (as conda-forge made this migration over a month ago).

The even better answer for htslib may be to split the plugin executables off into a separate htslib-plugins subpackage that the main htslib package would not depend on. This would mean that htslib/samtools/bcftools no longer depended on openssl at all, reducing their maintenance burden and immediately sidestepping all the historical openssl mismatch user issues. However it would also mean that people wanting to access files over HTTPS, S3, GCS, etc directly would need to be taught that they also need to install htslib-plugins — which may make this approach a non-starter.

jmarshall on Mar 5, 2023

I think the main source of the problems are old packages that have incomplete pinnings of their dependencies (in particular openssl dependencies). The proper way to fix them is to patch their metadata. This works via so-called repodata patching. The good thing: you can help with that.

Since our internal hackathon this tuesday, the process is now nicely described here: http://bioconda.github.io/developer/repodata_patching.html Hence, you just need to follow that guideline and introduce a small piece of fixing code for each old package that contains an unconstrained openssl dependency.

johanneskoester on Mar 3, 2023

At present, bioconda_utils pins conda-forge-pinning=2022.08.25.15.20.42.

Conda-forge moved to openssl 3 in conda-forge/conda-forge-pinning-feedstock#3892, merged on January 25th. So to move to rebuilding OpenSSL-using bioconda packages similarly, we will need to update bioconda_utils to pin a version of conda-forge-pinning past 2023.01.25.21.47.14 or so, and probably build updates on the bulk branch.

jmarshall on Feb 15, 2023