conan: [bug] concurrency errors in conan export

Environment Details (include every applicable attribute)

Operating System+version: macOS 12.3
Compiler+version: Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Conan version: Conan version 2.0.0-alpha7
Python version: Python 3.7.1

Steps to reproduce (Include if Applicable)

launch a simple script:

import subprocess
import multiprocessing

text = """from conan import ConanFile
class MyConanFile(ConanFile):
    pass
"""

with open("conanfile.py", "w") as f:
    f.write(text)

def run():
    while True:
        try:
            subprocess.check_call(["conan", "export", ".", "--name", "mylib", "--version", "1.0"])
        except:
            pass

jobs = []
for i in range(0, 2):
    process = multiprocessing.Process(target=run)
    jobs.append(process)
for j in jobs:
    j.start()
for j in jobs:
    j.join()

Logs (Executed commands with output) (Include/Attach if Applicable)

observe various random errors:

Traceback (most recent call last):
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/shutil.py", line 557, in move
    os.rename(src, real_dst)
OSError: [Errno 66] Directory not empty: '/Users/sse4/.conan2/p/tmp/e9336c83458230ec' -> '/Users/sse4/.conan2/p/0878a4b138b80085'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/cli/cli.py", line 163, in run
    command.run(self._conan_api, self._commands[command_argument].parser, args[0][1:])
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/cli/command.py", line 157, in run
    info = self._method(conan_api, parser, *args)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/cli/commands/export.py", line 39, in export
    lockfile=lockfile)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conan/api/subapi/__init__.py", line 21, in wrapper
    return f(subapi, *args, **kwargs)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conan/api/subapi/export.py", line 21, in export
    return cmd_export(app, path, name, version, user, channel, graph_lock=lockfile)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/cmd/export.py", line 62, in cmd_export
    cache.assign_rrev(recipe_layout)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conans/client/cache/cache.py", line 60, in assign_rrev
    return self._data_cache.assign_rrev(layout)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/site-packages/conan/cache/cache.py", line 224, in assign_rrev
    shutil.move(self._full_path(layout.base_folder), full_path)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/shutil.py", line 568, in move
    symlinks=True)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/shutil.py", line 315, in copytree
    os.makedirs(dst)
  File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/Users/sse4/.conan2/p/0878a4b138b80085'

 Traceback (most recent call last):
   File "/Users/sse4/.pyenv/versions/3.7.1/lib/python3.7/shutil.py", line 557, in move
    os.rename(src, real_dst)
OSError: [Errno 66] Directory not empty: '/Users/sse4/.conan2/p/tmp/b366d60a7e3773f7' -> '/Users/sse4/.conan2/p/0878a4b138b80085'

ERROR: Couldn't remove folder, might be busy or open: /Users/sse4/.conan2/p/0878a4b138b80085

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (10 by maintainers)

Most upvoted comments

Oh, yes, the commands are not designed to be directly called from other commands, that is what could be causing that --format doesn’t work. That private _method would be another possible point of breaking, but if it works for you in the meantime and you are aware of the risk and ready to fix it if a future 2.X release breaks it, then, it might be worth, it is just a few lines, not hundreds of them to fix.

memsharded on Mar 7, 2023

Thanks for the feedback, seems a reasonable approach

I like the idea of a global lock over a cache, that would be quite safe, and avoid difficult to debug race conditions. But I am also afraid that some users will inevitably complain, because they have their own usage patterns, and they know that building 2 different packages in parallel is safe for them…

memsharded on Mar 7, 2023

Conan 1.X never promised nor documented to be concurrency-safe, so in that sense it cannot be a regression.

Here you can find the summary for 2.0 that indicates the cache is not concurrent: https://github.com/conan-io/docs/pull/2593

memsharded on Jun 19, 2022