pyfakefs: INTERNALERROR /path/to/file is not in the subpath of '\\/path/to/file'

Describe the bug

In unit testing, I need to emulate a Windows OS. My base machine is macOS.

When using pyfakefs.fake_filesystem.OSType.WINDOWS + tempfile + --log-cli-level=DEBUG, I get the below internal error.

How To Reproduce

import tempfile

import pyfakefs.fake_filesystem


def test_foo(fs) -> None:
    fs.os = pyfakefs.fake_filesystem.OSType.WINDOWS

    with tempfile.TemporaryDirectory():
        pass

Outputs the following:

> pytest test/repo/test/test_play.py -s --log-cli-level=DEBUG
=============================================================================================================== test session starts ===============================================================================================================
platform darwin -- Python 3.10.9, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/user/code/repo/test, configfile: pyproject.toml
plugins: pyfakefs-5.2.2, mock-3.10.0, subtests-0.10.0
collected 1 item

test/repo/test/test_play.py::test_foo
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/runner.py", line 112, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/runner.py", line 131, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     hook.pytest_runtest_logreport(report=report)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/terminal.py", line 551, in pytest_runtest_logreport
INTERNALERROR>     line = self._locationline(rep.nodeid, *rep.location)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/terminal.py", line 878, in _locationline
INTERNALERROR>     res = mkrel(nodeid)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/terminal.py", line 868, in mkrel
INTERNALERROR>     line = self.config.cwd_relative_nodeid(nodeid)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1104, in cwd_relative_nodeid
INTERNALERROR>     nodeid = bestrelpath(self.invocation_params.dir, fullpath)
INTERNALERROR>   File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/pathlib.py", line 710, in bestrelpath
INTERNALERROR>     reldirectory = directory.relative_to(base)
INTERNALERROR>   File "/Users/user/.pyenv/versions/3.10.9/lib/python3.10/pathlib.py", line 818, in relative_to
INTERNALERROR>     raise ValueError("{!r} is not in the subpath of {!r}"
INTERNALERROR> ValueError: '/Users/user/code/repo' is not in the subpath of '\\/Users/user/code/repo' OR one path is relative and the other is absolute.
Traceback (most recent call last):
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_filesystem.py", line 1618, in get_object_from_normpath
    target = target.get_entry(component)  # type: ignore
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_file.py", line 542, in get_entry
    return self.entries[to_string(pathname_name)]
KeyError: 'Users'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/code/repo/venv/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 190, in console_main
    code = main()
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/config/__init__.py", line 167, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/main.py", line 317, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/_pytest/main.py", line 302, in wrap_session
    os.chdir(session.startpath)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_os.py", line 1354, in wrapped
    return f(*args, **kwargs)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_os.py", line 421, in chdir
    self.filesystem.confirmdir(path)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_filesystem.py", line 2810, in confirmdir
    self.resolve(target_directory, check_owner=check_owner),
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_filesystem.py", line 1692, in resolve
    return self.get_object_from_normpath(
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_filesystem.py", line 1627, in get_object_from_normpath
    self.raise_os_error(errno.ENOENT, path)
  File "/Users/user/code/repo/venv/lib/python3.10/site-packages/pyfakefs/fake_filesystem.py", line 418, in raise_os_error
    raise OSError(err_no, message, filename)
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: 'C:\\Users\\user\\code\\repo'

Your environment

I have pyfakefs==5.2.2. Note the current version printing from the issue template is broken.

> python -c "import platform; print(platform.platform())"
macOS-12.6-arm64-arm-64bit
> python -c "import sys; print('Python', sys.version)"
Python 3.10.9 (main, Jan  6 2023, 11:19:20) [Clang 14.0.0 (clang-1400.0.29.102)]
> python -c "from pyfakefs.fake_filesystem import __version__; print('pyfakefs', __version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name '__version__' from 'pyfakefs.fake_filesystem' (/Users/user/proj/proj_venv/lib/python3.10/site-packages/pyfakefs/fake_filesystem.py)
> python -c "import pytest; print('pytest', pytest.__version__)"
pytest 7.2.2

About this issue

Original URL
State: closed
Created a year ago
Comments: 21 (20 by maintainers)

Commits related to this issue

Re-create the temp directory on resetting the filesystem - the temp dir is now created during filesystem initialization instead of at patcher setup - fixes #814 — committed to mrbean-bremen/pyfakefs by mrbean-bremen a year ago
Re-create the temp directory on resetting the filesystem - the temp dir is now created during filesystem initialization instead of at patcher setup - fixes #814 — committed to mrbean-bremen/pyfakefs by mrbean-bremen a year ago
Re-create the temp directory on resetting the filesystem - the temp dir is now created during filesystem initialization instead of at patcher setup if configured - this is only done by default if c... — committed to mrbean-bremen/pyfakefs by mrbean-bremen a year ago
Re-create the temp directory on resetting the filesystem (#818) * Re-create the temp directory on resetting the filesystem - the temp dir is now created during filesystem initialization instead... — committed to pytest-dev/pyfakefs by mrbean-bremen a year ago
Exclude pytest modules from patching - fixes some specific tests in namespace packages - fixes #814 — committed to mrbean-bremen/pyfakefs by mrbean-bremen a year ago
Exclude pytest pathlib module from patching (#828) - fixes some specific tests in namespace packages - fixes #814 — committed to pytest-dev/pyfakefs by mrbean-bremen a year ago

Most upvoted comments

I had another look, and it seems that this can indeed be prevented by excluding some pytest modules from patching. I’m not completely sure about possible side effects, but I think that these modues shall never be patched. You can test this by modifying your test like this:

import pyfakefs.fake_filesystem
from pyfakefs.fake_filesystem_unittest import Patcher
from _pytest import capture, pathlib

Patcher.SKIPMODULES.add(capture)
Patcher.SKIPMODULES.add(pathlib)


def test_foo(fs) -> None:
    fs.os = pyfakefs.fake_filesystem.OSType.WINDOWS

The additional skip modules would be added to the pytest plugin, but you can test it locally this way. Please let me know if this fixes the problem for you!

mrbean-bremen on May 16, 2023

Thanks for the effort! I will have a closer look as soon as I find the time - probably at the end of this week. This also sounds like some pytest-related problem, e.g. something in the pytest code got patched that shouldn’t - I will see if there is an easy solution (like adding something to skipped modules).

mrbean-bremen on May 15, 2023

Yes, I think you are right, and the problem here is indeed related to the mixup of different filesystem paths. The thing is that the emulation of another OS in pyfakefs is not perfect anyway - there are more problems related to pathlib, and you can’t completely rely on the behavior to be the same as on the native OS. I may have to revisit the related code and think about how to make this better - this may take a bit, though…

mrbean-bremen on Apr 25, 2023

Ok, I think I know what the problem is - I think this is actually a regression. On resetting the file system on changing the OS the temp is not recreated, which seem to cause the problem. I will see if I can think ths in the next days.

mrbean-bremen on Apr 18, 2023

Thanks for the report! The cross-os functionality of pyfakefs is not perfect, and mostly only used in our own tests, so there are certainly issues. In this case it has probably to do with the different location of temp directories, though I have to have a closer look later.

As for the broken issue template - I actually noticed this in the last bug report, but forgot to check it out. Thanks for the reminder!

mrbean-bremen on Apr 18, 2023