mamba: Reproducible micromamba segfault when installing from lockfile after previous install

I can reproduce in a Dockerfile built on GitHub Actions.

I put some effort into making this more minimal, but didn’t get very far. I hope this makes it possible for someone to easily do a trace. 😃

High-level overview:

Select some particular dependencies
Do micromamba install, with or without lockfile. (I used a lockfile for the first install command to guarantee reproducibility.)
Do micromamba install from a new-style lockfile produced by conda-lock

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 57 (39 by maintainers)

Commits related to this issue

Merge pull request #1893 from Klaim/klaim/fix-env-lockfile-install-crash Fixed: crash when installing with env-lockfile (fixes #1882) — committed to tl-hbk/mamba by JohanMabille 2 years ago

Most upvoted comments

It is the commit that introcuced the lock file feature in the first place. So I ran a bisect with 1 step 😄

jonashaag on Aug 30, 2022

Probably the language @wolfv was most familiar with, which is a very good reason 😃

jonashaag on Aug 29, 2022

Livestream!

This worked for me to enable debug symbols:

cmake -DCMAKE_BUILD_TYPE=Debug

Culprit:

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x6422203a2236354a)
    frame #0: 0x000000010080abc8 libmamba.2.dylib`mamba::PackageInfo::PackageInfo(this=0x000000016fdfda90, s=0x000000010580d270) at package_info.cpp:94:31 [opt]
   91  	    PackageInfo::PackageInfo(Solvable* s)
   92  	    {
   93  	        // Note: this function (especially the checksum part) is NOT YET threadsafe!
-> 94  	        Pool* pool = s->repo->pool;
   95  	        const char* str;

I like to use ASan to debug these kinds of things. Luckily Mamba already has a compile flag for that

-DENABLE_ASAN=ON

Output:

=================================================================
==18052==ERROR: AddressSanitizer: container-overflow on address 0x00010bd023b7 at pc 0x0001061254a4 bp 0x00016b0f1e70 sp 0x00016b0f1e68
READ of size 1 at 0x00010bd023b7 thread T0
    #0 0x1061254a0 in mamba::Configuration::add_to_loading_sequence(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&) vector
    #1 0x106123ee8 in mamba::Configuration::compute_loading_sequence() configuration.cpp:1699
    #2 0x106122740 in mamba::Configuration::load() configuration.cpp:1658
    #3 0x1061e0c50 in mamba::install() install.cpp:344
    #4 0x104d5cdc0 in CLI::App::run_callback(bool, bool) App.hpp:1955
    #5 0x104d5bc5c in CLI::App::parse(int, char const* const*) App.hpp:1223
    #6 0x104d93b8c in main main.cpp:88
    #7 0x105175088 in start+0x204 (dyld:arm64+0x5088)
    #8 0xcb107ffffffffffc  (<unknown module>)

0x00010bd023b7 is located 1847 bytes inside of 3072-byte region [0x00010bd01c80,0x00010bd02880)
allocated by thread T0 here:
    #0 0x106758018 in wrap_malloc+0x88 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x40018)
    #1 0x10533079c in operator new(unsigned long)+0x18 (libc++.1.0.dylib:arm64+0x1479c)
    #2 0x105d0f1fc in void std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::__push_back_slow_path<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) vector:1517
    #3 0x10611c890 in mamba::Configuration::insert(mamba::Configurable, bool) configuration.hpp:883
    #4 0x1060fe3b8 in mamba::Configuration::set_configurables() configuration.cpp:1491
    #5 0x1060e6da0 in mamba::Configuration::Configuration() configuration.cpp:965
    #6 0x105ca58d8 in void std::__1::__call_once_proxy<std::__1::tuple<mamba::singletons::Singleton<mamba::Configuration>& mamba::singletons::init_once<mamba::singletons::Singleton<mamba::Configuration>, std::__1::default_delete<mamba::singletons::Singleton<mamba::Configuration> > >(std::__1::unique_ptr<mamba::singletons::Singleton<mamba::Configuration>, std::__1::default_delete<mamba::singletons::Singleton<mamba::Configuration> > >&)::'lambda'()&&> >(void*) mutex:651
    #7 0x105330684 in std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*))+0x8c (libc++.1.0.dylib:arm64+0x14684)
    #8 0x105ca240c in mamba::singletons::Singleton<mamba::Configuration>& mamba::singletons::init_once<mamba::singletons::Singleton<mamba::Configuration>, std::__1::default_delete<mamba::singletons::Singleton<mamba::Configuration> > >(std::__1::unique_ptr<mamba::singletons::Singleton<mamba::Configuration>, std::__1::default_delete<mamba::singletons::Singleton<mamba::Configuration> > >&) singletons.cpp:123
    #9 0x104d83310 in init_rc_options(CLI::App*) common_options.cpp:17
    #10 0x104d838f0 in init_general_options(CLI::App*) common_options.cpp:39
    #11 0x104db9b44 in set_umamba_command(CLI::App*) umamba.cpp:29
    #12 0x104d93a2c in main main.cpp:46
    #13 0x105175088 in start+0x204 (dyld:arm64+0x5088)
    #14 0xcb107ffffffffffc  (<unknown module>)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow vector in mamba::Configuration::add_to_loading_sequence(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >&)
Shadow bytes around the buggy address:
  0x0070217c0420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0070217c0430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0070217c0440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0070217c0450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0070217c0460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0070217c0470: 00 00 00 00 fc fc[fc]fc fc fc fc fc fc fc fc fc
  0x0070217c0480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x0070217c0490: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x0070217c04a0: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x0070217c04b0: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x0070217c04c0: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==18052==ABORTING
fish: Job 1, 'micromamba/micromamba install -…' terminated by signal SIGABRT (Abort)

Actually this looks harmless so I’m following the recommendation to use ASAN_OPTIONS=detect_container_overflow=:

==18104==ERROR: AddressSanitizer: heap-use-after-free on address 0x00010ba00180 at pc 0x000105521018 bp 0x00016baca490 sp 0x00016baca488
READ of size 8 at 0x00010ba00180 thread T0
    #0 0x105521014 in mamba::MTransaction::fetch_extract_packages() transaction.cpp:1123
    #1 0x10551bc50 in mamba::MTransaction::execute(mamba::PrefixData&) transaction.cpp:928
    #2 0x105739bf8 in mamba::install_lockfile_specs(fs::u8path const&, bool) install.cpp:582
    #3 0x105738f3c in mamba::install() install.cpp:353
    #4 0x104380dc0 in CLI::App::run_callback(bool, bool) App.hpp:1955
    #5 0x10437fc5c in CLI::App::parse(int, char const* const*) App.hpp:1223
    #6 0x1043b7b8c in main main.cpp:88
    #7 0x1045cd088 in start+0x204 (dyld:arm64+0x5088)
    #8 0xa56a7ffffffffffc  (<unknown module>)

0x00010ba00180 is located 128 bytes inside of 14336-byte region [0x00010ba00100,0x00010ba03900)
freed by thread T0 here:
    #0 0x105cb0284 in wrap_realloc+0x90 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x40284)
    #1 0x104a01840 in solv_extend_realloc+0x7c (libsolv.1.dylib:arm64+0x39840)
    #2 0x1049e9f2c in pool_add_solvable+0x2c (libsolv.1.dylib:arm64+0x21f2c)
    #3 0x1049ef7b0 in repo_add_solvable+0x14 (libsolv.1.dylib:arm64+0x277b0)
    #4 0x1054bebe8 in mamba::MRepo::add_package_info(s_Repodata*, mamba::PackageInfo const&) repo.cpp:155
    #5 0x1054bf8b8 in mamba::MRepo::MRepo(mamba::MPool&, mamba::PrefixData const&) repo.cpp:84
    #6 0x1054c19d0 in mamba::MRepo::create(mamba::MPool&, mamba::PrefixData const&) repo.cpp:290
    #7 0x105739b54 in mamba::install_lockfile_specs(fs::u8path const&, bool) install.cpp:582
    #8 0x105738f3c in mamba::install() install.cpp:353
    #9 0x104380dc0 in CLI::App::run_callback(bool, bool) App.hpp:1955
    #10 0x10437fc5c in CLI::App::parse(int, char const* const*) App.hpp:1223
    #11 0x1043b7b8c in main main.cpp:88
    #12 0x1045cd088 in start+0x204 (dyld:arm64+0x5088)
    #13 0xa56a7ffffffffffc  (<unknown module>)

previously allocated by thread T0 here:
    #0 0x105cb0018 in wrap_malloc+0x88 (libclang_rt.asan_osx_dynamic.dylib:arm64+0x40018)
    #1 0x104a01858 in solv_extend_realloc+0x94 (libsolv.1.dylib:arm64+0x39858)
    #2 0x1049e99e0 in pool_create+0x6c (libsolv.1.dylib:arm64+0x219e0)
    #3 0x10542726c in mamba::MPool::MPool() pool.cpp:49
    #4 0x1057397f4 in mamba::install_lockfile_specs(fs::u8path const&, bool) install.cpp:582
    #5 0x105738f3c in mamba::install() install.cpp:353
    #6 0x104380dc0 in CLI::App::run_callback(bool, bool) App.hpp:1955
    #7 0x10437fc5c in CLI::App::parse(int, char const* const*) App.hpp:1223
    #8 0x1043b7b8c in main main.cpp:88
    #9 0x1045cd088 in start+0x204 (dyld:arm64+0x5088)
    #10 0xa56a7ffffffffffc  (<unknown module>)

SUMMARY: AddressSanitizer: heap-use-after-free transaction.cpp:1123 in mamba::MTransaction::fetch_extract_packages()
Shadow bytes around the buggy address:
  0x00702175ffe0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x00702175fff0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007021760000: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007021760010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007021760020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x007021760030:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007021760040: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007021760050: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007021760060: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007021760070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007021760080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==18104==ABORTING
fish: Job 1, 'ASAN_OPTIONS=detect_container_o…' terminated by signal SIGABRT (Abort)

This looks much more suspicious. Use after free, responsible for almost all security vulnerabilities in C/C++ code 😃

I wonder if this is a bug in libsolv. No, doesn’t work with older versions either.

I’m not deep enough into libsolv terrain to understand what’s wrong here. Starting another git bisect…

Bug exists in micromamba-0.24.0.

Now testing 5c41be2f8239a40b4456fdaf86db6d1b367994d2 which added the feature.

jonashaag on Aug 29, 2022