conan: [bug] access denied while renaming unzipped folders

In another context I’ve observed the permission denied error and there I’ve tried to isolate the problem. With the following python script I was able to produce the error nearly 100%:

import errno
import os
import platform
import shutil
import stat
import time


def isWindows():
  """ Returns True when running with the native port of Python for Windows,
  False when running on any other platform (including the Cygwin port of
  Python).
  """
  # Note: The cygwin port of Python returns "CYGWIN_NT_xxx"
  return platform.system() == "Windows"


def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      else:
        raise
  else:
    os.rename(src, dst)


def rmtree(path, ignore_errors=False):
  """shutil.rmtree(path) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  onerror = None
  if isWindows():
    path = path
    onerror = handle_rmtree_error
  shutil.rmtree(path, ignore_errors=ignore_errors, onerror=onerror)


def handle_rmtree_error(function, path, excinfo):
  # Allow deleting read-only files
  os.chmod(path, stat.S_IWRITE)
  function(path)


def rename_test():
  if os.path.isdir("old_dir"):
    rmtree("old_dir")
  os.mkdir("old_dir")
  for i in range(100):
    file = "old_dir/test_" + str(i).zfill(3) + ".txt"
    #print("Generate " + file)
    with open(file,"w") as fp:
      for n in range(100):
        fp.write(str(i).zfill(3) + "." + str(n).zfill(3) + ": 01234567890 01234567890 0123456789\n")
  print("Rename old_dir => new_dir")
  rename("old_dir", "new_dir")
  rmtree("new_dir")


# Problem occured for: D:/ma/pkg/baseswp\\.git.tmp' -> 'D:/ma/pkg/baseswp\\.git
for i in range(100):
  print("Rename test " + str(i))
  rename_test()

• I’ve run the tests also in a directory where no Windows indexing service is looking into => error still occurs
• I’ve checked the Virus Scanner logs, but could not find an entry

=> So my conclusion so far is that maybe the file operations (before the call to os.rename()) return already before the file handle is released (async.).

As a workaround I’ve implemented a retry loop with waiting 500 ms between retries:

def rename(src, dst):
  """os.rename(src, dst) wrapper with support for long paths on Windows.

  Availability: Unix, Windows."""
  if isWindows():
    # On Windows, rename fails if destination exists, see
    # https://docs.python.org/2/library/os.html#os.rename
    try:
      os.rename(src, dst)
    except OSError as e:
      if e.errno == errno.EEXIST:
        os.remove(dst)
        os.rename(src, dst)
      elif e.errno == errno.EACCES:
        # Workaround for the sporadic problem on Windows:
        # PermissionError: [WinError 5] Access denied
        # Retry 3 times to rename with sleeping 500 ms in between
        retry = 3
        while retry:
          #print("*** Retry: " + str(4 - retry))
          time.sleep(0.5)
          try:
            os.rename(src, dst)
            retry = 0
          except OSError as e:
            if retry and e.errno == errno.EACCES:
              retry = retry - 1
            else:
              raise
      else:
        raise
  else:
    os.rename(src, dst)

I’ve checked again a conan create with physx and antivirus disabled (on my personnal computer, can’t do that on corporate desktop), I don’t have permission denied error in this case.

@yangcha please, check #6774 people reviewed there but you didn’t answer.

Sorry. I thought the PR was rejected. I will answer that and resolve the conflicts .

A couple of things:

Windows indexing is disabled for .conan folder

As pointed in the other issue, maybe the AntiVirus could be related as well? Is it excluding the cache?

@uilianries Can you please try to reproduce with those recipes in Windows? The try-sleep sounds too much of a workaround, I’d prefer to know a bit better the reasons of this failure and see if something else could be done.