conan: [bug] Setting `core.download:parallel` causes `ERROR: database is locked`

Environment details

Operating System+version: Windows Server 2016, Ubuntu 20.04
- All machines have >= 48 vCPUs
Compiler+version: N/A
Conan version: 2.0.4, 2.0.5
Python version: 3.11

Steps to reproduce

Set core.download:parallel to some value. In our case, this is global.conf:

core:non_interactive = True
core.download:parallel = 10

Invoke conan install ...
sqlite3.OperationalError: database is locked

I should note that conan install is invoked in succession by a Python script. However, no amount of sleep() seems to fix the issue and unsetting core.download:parallel makes the problem disappear

Logs

conan install, given a standard conanfile.py and paths to host and build profiles

2023-05-18T18:19:35.7639220Z -------- Downloading 13 packages --------
2023-05-18T18:19:35.7639762Z Downloading binary packages in 10 parallel threads
2023-05-18T18:19:35.7678332Z fmt/9.1.0: Retrieving package 6c6677cbc81464b6b6b1695787db778b0be09550 from remote '[REDACTED]' 
2023-05-18T18:19:35.7679416Z imgui/1.87: Retrieving package 324b368d774fa34a6cdb068fe2ef8192a818cc88 from remote '[REDACTED]' 
2023-05-18T18:19:35.7680340Z json-c/0.15: Retrieving package b9bf0ccbd7c2de257a9ab3cfb4dd16ed65007983 from remote '[REDACTED]' 
2023-05-18T18:19:35.7683493Z libjpeg-turbo/2.1.5@[REDACTED]: Retrieving package 27567f108b4464edaf4634e2589c286d9b0cfd85 from remote '[REDACTED]' 
2023-05-18T18:19:35.7684566Z libiconv/1.17: Retrieving package b9bf0ccbd7c2de257a9ab3cfb4dd16ed65007983 from remote '[REDACTED]' 
2023-05-18T18:19:35.7687209Z dawn/5579@[REDACTED]: Retrieving package 1420f5eb1fd84f274aefcb2a9adeeaefff9d3dea from remote '[REDACTED]' 
2023-05-18T18:19:35.7689954Z libpng/1.6.39: Retrieving package 8e1ee4604f7dcf4947d3e06571faa4a7b589fda5 from remote '[REDACTED]' 
2023-05-18T18:19:35.7690762Z pcre/8.45: Retrieving package e26935b6bdd917b706a0a17334d607698d470cce from remote '[REDACTED]' 
2023-05-18T18:19:35.7694146Z liquidfun/1.1.0@[REDACTED]/dev: Retrieving package 324b368d774fa34a6cdb068fe2ef8192a818cc88 from remote '[REDACTED]' 
2023-05-18T18:19:35.7695056Z implot/0.14@[REDACTED]: Retrieving package ede40434c0b2ae7401db6ecf750d64a4edd6cd14 from remote '[REDACTED]' 
2023-05-18T18:19:35.8830595Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/fmt/9.1.0/_/_/revisions/e747928f85b03f48aaf227ff897d9634/packages/6c6677cbc81464b6b6b1695787db778b0be09550/revisions/79ed374e39171ec952a012f4579051e5/files/conan_package.tgz
2023-05-18T18:19:35.8837174Z Decompressing conan_package.tgz
2023-05-18T18:19:35.8920136Z fmt/9.1.0: Package installed 6c6677cbc81464b6b6b1695787db778b0be09550
2023-05-18T18:19:35.8920790Z fmt/9.1.0: Downloaded package revision 79ed374e39171ec952a012f4579051e5
2023-05-18T18:19:35.8921717Z libzip/1.8.0: Retrieving package c063fc064e9a56ccff6fb501a6d5e5ac74bba67b from remote '[REDACTED]' 
2023-05-18T18:19:36.0107439Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/imgui/1.87/_/_/revisions/5633ebf337730424e894d34203b11b6d/packages/324b368d774fa34a6cdb068fe2ef8192a818cc88/revisions/16d69ff4ccec8fd0ebf8fbb1c78d0d85/files/conan_package.tgz
2023-05-18T18:19:36.0184636Z Decompressing conan_package.tgz
2023-05-18T18:19:36.0484713Z imgui/1.87: Package installed 324b368d774fa34a6cdb068fe2ef8192a818cc88
2023-05-18T18:19:36.0485363Z imgui/1.87: Downloaded package revision 16d69ff4ccec8fd0ebf8fbb1c78d0d85
2023-05-18T18:19:36.0486245Z sdl/2.26.1@[REDACTED]: Retrieving package 4498fafa351ccb53920f61e14181f0b3c16806a5 from remote '[REDACTED]' 
2023-05-18T18:19:36.1200095Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/pcre/8.45/_/_/revisions/563d050c69aaa0a61be46bb09ba5c23c/packages/e26935b6bdd917b706a0a17334d607698d470cce/revisions/e7ec9bddd44826e0057df8f2a29403c9/files/conan_package.tgz
2023-05-18T18:19:36.1214034Z Decompressing conan_package.tgz
2023-05-18T18:19:36.1305277Z pcre/8.45: Package installed e26935b6bdd917b706a0a17334d607698d470cce
2023-05-18T18:19:36.1305906Z pcre/8.45: Downloaded package revision e7ec9bddd44826e0057df8f2a29403c9
2023-05-18T18:19:36.1306752Z freetype/2.13.0: Retrieving package 55351983b5f5d042a5022c172c89e30573d3f532 from remote '[REDACTED]' 
2023-05-18T18:19:36.2918597Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/sdl/2.26.1/[REDACTED]/revisions/97757243b40f008e925d55259af595de/packages/4498fafa351ccb53920f61e14181f0b3c16806a5/revisions/82ce4d7d7e0958818a934da63e291dce/files/conan_package.tgz
2023-05-18T18:19:36.2960636Z Decompressing conan_package.tgz
2023-05-18T18:19:36.3389210Z sdl/2.26.1@[REDACTED]: Package installed 4498fafa351ccb53920f61e14181f0b3c16806a5
2023-05-18T18:19:36.3389905Z sdl/2.26.1@[REDACTED]: Downloaded package revision 82ce4d7d7e0958818a934da63e291dce
2023-05-18T18:19:36.5695960Z Decompressing conan_package.tgz
2023-05-18T18:19:36.5745769Z json-c/0.15: Package installed b9bf0ccbd7c2de257a9ab3cfb4dd16ed65007983
2023-05-18T18:19:36.5746849Z json-c/0.15: Downloaded package revision 6c9a9340ebae4cf0e4994f5a7d527ac6
2023-05-18T18:19:36.6803579Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/freetype/2.13.0/_/_/revisions/a312c096b314c6cfd6b43e290c88928a/packages/55351983b5f5d042a5022c172c89e30573d3f532/revisions/a524e1f748840e13098a6c271742435b/files/conan_package.tgz
2023-05-18T18:19:36.6831777Z Decompressing conan_package.tgz
2023-05-18T18:19:36.7082534Z freetype/2.13.0: Package installed 55351983b5f5d042a5022c172c89e30573d3f532
2023-05-18T18:19:36.7083364Z freetype/2.13.0: Downloaded package revision a524e1f748840e13098a6c271742435b
2023-05-18T18:19:36.8712488Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/libiconv/1.17/_/_/revisions/fa54397801cd96911a8294bc5fc76335/packages/b9bf0ccbd7c2de257a9ab3cfb4dd16ed65007983/revisions/7caa9e5b3af799bceffcdbe1ee870ece/files/conan_package.tgz
2023-05-18T18:19:36.8781076Z Decompressing conan_package.tgz
2023-05-18T18:19:36.8931988Z libiconv/1.17: Package installed b9bf0ccbd7c2de257a9ab3cfb4dd16ed65007983
2023-05-18T18:19:36.8932709Z libiconv/1.17: Downloaded package revision 7caa9e5b3af799bceffcdbe1ee870ece
2023-05-18T18:19:37.0289862Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/libpng/1.6.39/_/_/revisions/a4953955f060811c79abff9fe986b6fe/packages/8e1ee4604f7dcf4947d3e06571faa4a7b589fda5/revisions/b3aaf7d1d260a23aaa89cd37dabf8768/files/conan_package.tgz
2023-05-18T18:19:37.0308358Z Decompressing conan_package.tgz
2023-05-18T18:19:37.0377380Z libpng/1.6.39: Package installed 8e1ee4604f7dcf4947d3e06571faa4a7b589fda5
2023-05-18T18:19:37.0379062Z libpng/1.6.39: Downloaded package revision b3aaf7d1d260a23aaa89cd37dabf8768
2023-05-18T18:19:37.1878249Z $Downloading conan_package.tgz from ***/api/conan/conan-internal/v2/conans/libjpeg-turbo/2.1.5/[REDACTED]/revisions/6fa0c11b0f58c7102a4e73003c6e76e1/packages/27567f108b4464edaf4634e2589c286d9b0cfd85/revisions/a4ee91372a20134eb537382e47732b91/files/conan_package.tgz
2023-05-18T18:19:37.1901411Z Decompressing conan_package.tgz
2023-05-18T18:19:37.2006714Z libjpeg-turbo/2.1.5@[REDACTED]: Package installed 27567f108b4464edaf4634e2589c286d9b0cfd85
2023-05-18T18:19:37.2008866Z libjpeg-turbo/2.1.5@[REDACTED]: Downloaded package revision a4ee91372a20134eb537382e47732b91
2023-05-18T18:19:37.2032496Z ERROR: database is locked
2023-05-18T18:19:37.2033015Z Traceback (most recent call last):
2023-05-18T18:19:37.2033753Z   File "/usr/local/lib/python3.10/dist-packages/conan/cli/cli.py", line 271, in main
2023-05-18T18:19:37.2034308Z     cli.run(args)
2023-05-18T18:19:37.2034934Z   File "/usr/local/lib/python3.10/dist-packages/conan/cli/cli.py", line 171, in run
2023-05-18T18:19:37.2035593Z     command.run(self._conan_api, self._commands[command_argument].parser, args[0][1:])
2023-05-18T18:19:37.2036330Z   File "/usr/local/lib/python3.10/dist-packages/conan/cli/command.py", line 134, in run
2023-05-18T18:19:37.2036931Z     info = self._method(conan_api, parser, *args)
2023-05-18T18:19:37.2037670Z   File "/usr/local/lib/python3.10/dist-packages/conan/cli/commands/install.py", line 84, in install
2023-05-18T18:19:37.2038362Z     conan_api.install.install_binaries(deps_graph=deps_graph, remotes=remotes)
2023-05-18T18:19:37.2039117Z   File "/usr/local/lib/python3.10/dist-packages/conan/api/subapi/install.py", line 22, in install_binaries
2023-05-18T18:19:37.2039747Z     installer.install(deps_graph, remotes)
2023-05-18T18:19:37.2040446Z   File "/usr/local/lib/python3.10/dist-packages/conans/client/installer.py", line 247, in install
2023-05-18T18:19:37.2041165Z     self._download_bulk(install_order)
2023-05-18T18:19:37.2041875Z   File "/usr/local/lib/python3.10/dist-packages/conans/client/installer.py", line 277, in _download_bulk
2023-05-18T18:19:37.2042490Z     thread_pool.map(self._download_pkg, downloads)
2023-05-18T18:19:37.2043066Z   File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
2023-05-18T18:19:37.2043884Z     return self._map_async(func, iterable, mapstar, chunksize).get()
2023-05-18T18:19:37.2044749Z   File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
2023-05-18T18:19:37.2045243Z     raise self._value
2023-05-18T18:19:37.2045768Z   File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
2023-05-18T18:19:37.2046317Z     result = (True, func(*args, **kwds))
2023-05-18T18:19:37.2046864Z   File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
2023-05-18T18:19:37.2047381Z     return list(map(*args))
2023-05-18T18:19:37.2048070Z   File "/usr/local/lib/python3.10/dist-packages/conans/client/installer.py", line 288, in _download_pkg
2023-05-18T18:19:37.2048758Z     self._remote_manager.get_package(node.conanfile, node.pref, node.binary_remote)
2023-05-18T18:19:37.2049550Z   File "/usr/local/lib/python3.10/dist-packages/conans/client/remote_manager.py", line 105, in get_package
2023-05-18T18:19:37.2050153Z     pkg_layout = self._cache.get_or_create_pkg_layout(pref)
2023-05-18T18:19:37.2050913Z   File "/usr/local/lib/python3.10/dist-packages/conans/client/cache/cache.py", line 87, in get_or_create_pkg_layout
2023-05-18T18:19:37.2051571Z     return self._data_cache.get_or_create_pkg_layout(ref)
2023-05-18T18:19:37.2052320Z   File "/usr/local/lib/python3.10/dist-packages/conan/internal/cache/cache.py", line 127, in get_or_create_pkg_layout
2023-05-18T18:19:37.2052932Z     return self.get_package_layout(pref)
2023-05-18T18:19:37.2053645Z   File "/usr/local/lib/python3.10/dist-packages/conan/internal/cache/cache.py", line 107, in get_package_layout
2023-05-18T18:19:37.2054256Z     pref_data = self._db.try_get_package(pref)
2023-05-18T18:19:37.2055004Z   File "/usr/local/lib/python3.10/dist-packages/conan/internal/cache/db/cache_database.py", line 73, in try_get_package
2023-05-18T18:19:37.2055621Z     ref_data = self._packages.get(ref)
2023-05-18T18:19:37.2056294Z   File "/usr/local/lib/python3.10/dist-packages/conan/internal/cache/db/packages_table.py", line 62, in get
2023-05-18T18:19:37.2056863Z     r = conn.execute(query)
2023-05-18T18:19:37.2057355Z sqlite3.OperationalError: database is locked

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 25 (23 by maintainers)

Most upvoted comments

Ok, I planned to do it later but will try today then.

vient on Oct 26, 2023

I think proper way is to first check that original script from python issue works, then modify it to have same code as in conan and see if it still shows database locking. I can try to do it later.

vient on Oct 20, 2023

I think because that is quite a difficult thing to understand and connect the dots to the conf when that is happening.

Who says that 20 seconds will always be enough? It will certainly be possible to make a scenario where this won’t be enough (e.g. extremely heavy load, slow disks, machine swapping like there is no tomorrow).

I think that if the system cannot sync a simple DB read/write in 20 seconds, then the problem is a different one, and the system is already degraded to a point when other things can also go wrong. But I get your point, let’s give it 20 seconds at this moment, and if it happens again we will consider the conf.

memsharded on Oct 19, 2023

We are still occasionally getting this error on our CI servers even with Conan 2.0.10 (haven’t tested with 2.0.11, but nothing from release notes indicates that there were any changes there).

DoDoENT on Sep 22, 2023

Then yeah, maybe the high-core count is the necessary thing to fire this, thanks for the feedback.

memsharded on May 19, 2023

No, no parallel jobs. We call this function sequentially for each build config (Debug, Release) and for two different conanfiles.

def _install_from_conanfile(
    path_to_conanfile: str,
    path_to_host_profile: str,
    path_to_build_profile: str,
    deploy_folder=None,
    environment=None,
    build_missing=False,
):
    conan_run_args = [
        "conan",
        "install",
        path_to_conanfile,
        f"--profile:host={path_to_host_profile}",
        f"--profile:build={path_to_build_profile}",
        "--generator=CMakeToolchain",
    ]

    if build_missing:
        conan_run_args.append("--build=missing")

    if deploy_folder:
        conan_run_args[-1:-1] = [
            "--deploy",
            "full_deploy",
            "--output-folder",
            deploy_folder,
        ]

    conan_res = subprocess.run(
        conan_run_args, env=environment if environment else os.environ,
    )

sykhro on May 19, 2023