tools-iuc: List of tools that fail containerized tool tests

Hello IUC,

Tl;DR:

Help us fix tool tests!

we would like to make sure that our tools are well tested and working in the widest range of deployment scenarios possible. And as you may have noticed, tool dependency installation from Conda may break (or be fixed!) as time passes due to dependency-of-a-dependency upgrades, newer/older Conda versions etc.

To make sure that our tool wrappers will work on a wide range of deployments it is important that we include all dependencies that a tool has (i.e include coreutils if a wrapper uses paste, include sed if a wrapper uses sed, include python if a wrapper calls a python script). One way to make sure this is the case is to run tool tests in a minimal container that does not contain any software except the one listed in our tool wrappers <requirements/> tag set.

We also currently build containers for all IUC tools as well as other tool repositories listed in https://github.com/galaxyproject/planemo-monitor, but so far we are not testing that tool runs actually succeed in these containers. For dependencies (and dependency combinations) that are not available as a downloadable container Galaxy can build containers when needed. This functionality is also available via planemo test --biocontainers.

We have set up a github workflow that tests all IUC tools using planemo test and the --biocontainers flag. The test run can be seen here: https://github.com/mvdbeek/tools-iuc/commit/529a26943046ca80b18bccae31021db5c317eb7e/checks?check_suite_id=331921644 And a html report of only the failing tests is here: https://mvdbeek.github.io/iuc-tool-test-results/ Note also that the tool tests run against python 3.7, so this could be another class of errors we may need to fix (should concern only the cheetah templating).

The list of tools with failing tests is the following:

• multigps @gregvonkuster - https://github.com/galaxyproject/tools-iuc/pull/2754
• ucsc-twobittofa @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2731
• umi_tools (pysam ImportError: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory) @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2729 and container rebuild
  • umi_tools_group
  • umi_tools_dedup
• scanpy: @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2704
  • scanpy_filter
  • scanpy_normalize
  • scanpy_cluster_reduce_dimension
  • scanpy_plot
  • scanpy_remove_confounders
  • scanpy_inspect
• ncbi_eutils_epost
• quast (UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 14588: ordinal not in range(128)) https://github.com/bioconda/bioconda-recipes/pull/19062
• anndata_inspect @mvdbeek needs extended-base container
• snpEff_get_chr_names
• art: @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2732
  • art_454
  • art_illumina
  • art_solid
• ebi_search_rest_results (HTTP Error 400: )
• ebi_metagenomics_run_downloader (ModuleNotFoundError: No module named ‘xmltramp2’)
• scpipe
• EMBOSS:_wobble108 (missing perl) @bernt-matthias https://github.com/galaxyproject/tools-iuc/pull/2725
• mykrobe_predict
• raceid_filtnormconf @mvdbeek
• repeatmasker_wrapper (most likely sed not working on header)
• heinz_visualization https://github.com/bioconda/bioconda-recipes/pull/19070
• dropletutils
• CoverageReport2 (perl: unable to load shared object ‘/usr/local/lib/R/library/grDevices/libs//cairo.so’: libXrender.so.1: cannot open shared object file: No such file or directory)
• busco (tar + maybe [makeblastdb] /usr/local/bin/makeblastdb: error while loading shared libraries: libbz2.so.1) https://github.com/galaxyproject/tools-iuc/pull/2715 https://github.com/bioconda/bioconda-recipes/pull/19016
• simple_weather
• ncbi_eutils_efetch
• onto_tk_get_relationship_id_vs_relationship_def
• migmap (blast: error while loading shared libraries: libbz2.so.1) https://github.com/bioconda/bioconda-recipes/pull/19084 https://github.com/galaxyproject/tools-iuc/pull/2738
• glimmer_acgt_content
• cshl_fastx_nucleotides_distribution
• snippy_core (tar unrecognized option ‘–strip-components=1’) https://github.com/galaxyproject/tools-iuc/pull/2716
• raceid_inspectclusters @mvdbeek
• mothur_chimera_slayer @nsoranzo / https://github.com/galaxyproject/tools-iuc/pull/2702
• glimmer_extract
• scikit_bio_diversity_beta_diversity
• breseq (tar -z) https://github.com/galaxyproject/tools-iuc/pull/2713
• bwa_mem_index_builder_data_manager @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2724
• cshl_fastq_quality_boxplot
• onto_tk_get_terms_by_relationship_type
• hicexplorer_hicplottads
• enasearch_retrieve_analysis_report
• codeml (mv: can’t remove ‘/tmp/tmpx34zwk08/job_working_directory/000/3/tmp5l4xkv9b’: Read-only file system)
• ncbi_eutils_einfo
• query_tabular (diff probably caused somewhere due to floating point precision)
• fasttree (docker: invalid reference format: repository name must be lowercase.) @mvdbeek
• gemini_burden
• poretools_squiggle
• pipelign
• stacks2 @bernt-matthias (diff with test data … don’t understand yet)
  • stacks2_ustacks
  • stacks2_denovomap
• stacks @bernt-matthias (all seem to be installation problems: command not found)
  • stacks_cstacks
  • stacks_clonefilter
  • stacks_ustacks
  • stacks_rxstacks
  • stacks_populations
  • stacks_denovomap
  • stacks_genotypes
  • stacks_sstacks
  • stacks_pstacks
  • stacks_refmap
  • stacks_assembleperead
  • stacks_procrad
• samtools @bernt-matthias (does not find test data for cached case because test-data/cached_locally is a symlink) https://github.com/galaxyproject/tools-iuc/pull/2712
  • samtools_bam_to_cram
  • samtools_cram_to_bam
  • samtools_view
• dada2 @bernt-matthias (all but fetcher are due to diff in produced Rdata files, fetcher real bug) https://github.com/galaxyproject/tools-iuc/pull/2705
  • dada2_mergePairs
  • dada2_removeBimeraDenovo
  • dada2_dada
  • dada2_learnErrors
  • dada2_fetcher
• maxbin2 @bernt-matthias (tar wo -z option) https://github.com/galaxyproject/tools-iuc/pull/2706
• mummer_mummerplot
• Extract_genomic_DNA_1 (faToTwoBit: error while loading shared libraries: libstdc++.so.6:) @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2731
• transtermhp (env variable from /etc/conda/activate.d/ not set) https://github.com/galaxyproject/tools-iuc/pull/2726
• snap_training
• raceid_clustering @mvdbeek
• glimmer_build_icm
• raxml (python: command not found) https://github.com/galaxyproject/tools-iuc/pull/2707
• gemini_annotate
• ruvseq (symlinked R script not found)
• sina
• meme_fimo - @gregvonkuster https://github.com/galaxyproject/tools-iuc/pull/2744
• fermi2
• raceid_inspecttrajectory @mvdbeek
• enasearch_retrieve_taxons
• miniasm (error while loading shared libraries: libz.so.1)
• star_fusion
• hicexplorer_hicpca
• glimmer_not_knowledge_based
• resize_coordinate_window @gregvonkuster https://github.com/galaxyproject/tools-iuc/pull/2745
• newick_display (convert: command not found) https://github.com/galaxyproject/tools-iuc/pull/2710
• poretools_events
• poretools_yield_plot
• trinotate
• taxonomy_krona_chart (/usr/bin/perl: bad interpreter: No such file or directory) https://github.com/galaxyproject/tools-iuc/pull/2709
• pathview (png output size diff)
• bcftools
  • bcftools_stats (env: can’t execute ‘perl’: No such file or directory)
  • bcftools_plugin_color_chrs (env: can’t execute ‘perl’: No such file or directory)
  • bcftools_plugin_counts (python: command not found)
• bctools
  • bctools_merge_pcr_duplicates (sort: unrecognized option ‘–compress-program=gzip’)
  • bctools_remove_spurious_events (probably missing perl or wrong sort usage)
• graphlan (conda conflict)
  • graphlan (graphlan.py: command not found)
  • graphlan_annotate.py: command not found
• graphlan_annotate
• cshl_fastx_barcode_splitter
• poretools_occupancy
• phyml
• enasearch_search_data
• bowtie2_index_builder_data_manager @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2723
• varscan_somatic
• bracken_build_database
• variant_analyzer @bernt-matthias
  • read2mut (matplotlib)
  • mut2read
  • mut2sscs
• jbrowse @abretaud
• abyss-pe @mvdbeek https://github.com/bioconda/bioconda-recipes/pull/19060
• prinseq (/usr/local/bin/perl shebang + recipe bump was forgotten) https://github.com/bioconda/bioconda-recipes/pull/19014
• kraken https://github.com/galaxyproject/tools-iuc/pull/2721
  • kraken (some docker problem)
  • kraken-report
  • kraken-translate
  • kraken-mpa-report
• kraken2_build_database (the kraken perl script uses @ in path Possible unintended interpolation of @2 in string at /home/berntm/miniconda3/envs/__kraken2@2.0.8_beta/bin/kraken2, needs conda fix)
• valet
• meme_meme - @gregvonkuster - https://github.com/galaxyproject/tools-iuc/pull/2744
• pe_histogram (fonts: libfreetype.so.6: cannot open shared object file, JAVA) https://github.com/galaxyproject/tools-iuc/pull/2728
• raceid_trajectory @mvdbeek
• vcffilter2
• column_order_header_sort
• Bandage: @mvdbeek https://github.com/galaxyproject/tools-iuc/pull/2733
  • bandage_info
  • bandage_image
• structure
• glimmer_long_orfs
• humann2
• rcorrector
• onto_tk_get_relationship_id_vs_relationship_name
• enasearch_retrieve_run_report
• feelnc @abretaud #2757
• prokka (version mismatch in test data and maybe tbl2asn to old?) https://github.com/galaxyproject/tools-iuc/pull/2717 still needs docker image rebuild
• snpEff_build_gb
• slamdunk
• hicexplorer_hicplotmatrix
• kraken-filter
• orthofinder_onlygroups @abretaud #2757
• prestor_abseq3
• augustus_training @abretaud #2759
• kissplice (error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory)
• onto_tk_get_relationship_types
• poretools_qualpos
• goenrichment (fonts, Java) https://github.com/bioconda/bioconda-recipes/pull/19043
• tb_profiler_profile
• kobas_annotate
• circos (tar -z AND mv: can’t remove ‘/tmp/tmpzvg40cuz/job_working_directory/000/7/tmparfo8udy’: Read-only file system) https://github.com/galaxyproject/tools-iuc/pull/2714
  • circos
  • circos_tableviewer
• bax2bam (mv: can’t remove ‘/tmp/tmpzvg40cuz/job_working_directory/000/2/tmpnuxq8_9j’: Read-only file system)
• htseq_count
• gfa_to_fa
• hmmer3 https://github.com/galaxyproject/tools-iuc/pull/2720
  • hmmer_nhmmer
  • hmmer_nhmmscan
• mykrobe_genotype
• idr
• masigpro https://github.com/galaxyproject/tools-iuc/pull/2722
• poretools_hist
• cshl_fastx_trimmer
• qualimap (Java + fonts) https://github.com/bioconda/bioconda-recipes/pull/19041
  • qualimap_multi_bamqc
  • qualimap_rnaseq
• snpEff_download
• glimmer_knowledge_based
• tag_pileup_frequency - https://github.com/galaxyproject/tools-iuc/pull/2727 - current failure: Applying linter tool_xsd… FAIL, The attribute ‘designation’ is not allowed
• qiime (uses python2.7 but errors on ‘getpwuid(): uid not found: 1001’) @mvdbeek needs writable home dir, done. Also needs extended base image, need to rewrite mulled-build
  • qiime_summarize_taxa_through_plots
  • qiime_alpha_rarefaction
  • qiime_upgma_cluster
  • qiime_extract_barcodes
  • qiime_alpha_diversity
  • qiime_compare_categories
  • qiime_split_libraries_fastq
  • qiime_beta_diversity
  • qiime_plot_taxa_summary
  • qiime_jackknifed_beta_diversity
  • qiime_pick_otus
  • qiime_summarize_taxa
  • qiime_validate_mapping_file
  • qiime_count_seqs
  • qiime_filter_otus_from_otu_table
  • qiime_filter_taxa_from_otu_table
  • qiime_pick_open_reference_otus
  • qiime_collapse_samples
  • qiime_beta_diversity_through_plots
  • qiime_multiple_split_libraries_fastq
  • qiime_filter_fasta
  • qiime_pick_closed_reference_otus
  • qiime_split_libraries
  • qiime_filter_samples_from_otu_table
  • qiime_make_emperor
  • qiime_assign_taxonomy
  • qiime_filter_alignment
  • qiime_make_otu_table
  • qiime_core_diversity
  • qiime_make_otu_heatmap
  • qiime_pick_rep_set
  • qiime_multiple_join_paired_ends
• fsd (ModuleNotFoundError: No module named ‘matplotlib’) https://github.com/galaxyproject/tools-iuc/pull/2718
  • fsd_regions (same)
  • sdsd_beforevsafter
  • td
• repmatch_gff3 (ModuleNotFoundError: No module named ‘matplotlib’) https://github.com/galaxyproject/tools-iuc/pull/2718

If you’re currently working on one of these tools write your name under the tool, so that we avoid duplicating our work