spack: Debian ruamel.yaml 0.15: Infinite Recoursion On Setup
Migrated out of https://github.com/spack/spack/issues/2495#issuecomment-419636297
Just moving it out not to clutter it further.
I currently get an infinite recursion on a fresh copy of spack develop (ccbff6e0cdc1e4e045656a784e28225f416a8ed4) and removed $HOME/.spack directory on . share/spack/setup_env.sh in Debian 9.5 “stretch”.
Steps to reproduce the issue
See docker image below. Basically, make sure to have a recent version of ruamel.yaml on your system.
Error Message
Traceback (most recent call last):
File "/home/axel/src/spack/bin/spack", line 54, in <module>
sys.exit(spack.main.main())
File "/home/axel/src/spack/lib/spack/spack/main.py", line 600, in main
print_setup_info(*args.print_shell_vars.split(','))
File "/home/axel/src/spack/lib/spack/spack/main.py", line 565, in print_setup_info
module_roots = spack.config.get('config:module_roots')
File "/home/axel/src/spack/lib/spack/spack/config.py", line 573, in get
return config.get(path, default, scope)
File "/home/axel/src/spack/lib/spack/llnl/util/lang.py", line 556, in __getattr__
return getattr(self.instance, name)
File "/home/axel/src/spack/lib/spack/llnl/util/lang.py", line 556, in __getattr__
return getattr(self.instance, name)
File "/home/axel/src/spack/lib/spack/llnl/util/lang.py", line 556, in __getattr__
return getattr(self.instance, name)
[Previous line repeated 318 more times]
File "/home/axel/src/spack/lib/spack/llnl/util/lang.py", line 552, in instance
self._instance = self.factory()
File "/home/axel/src/spack/lib/spack/spack/config.py", line 547, in _config
defaults = InternalConfigScope('_builtin', config_defaults)
File "/home/axel/src/spack/lib/spack/spack/config.py", line 238, in __init__
_validate_section({section: dsec}, section_schemas[section])
File "/home/axel/src/spack/lib/spack/spack/config.py", line 611, in _validate_section
_validate_section.validator(schema).validate(data)
File "/home/axel/src/spack/lib/spack/external/jsonschema/validators.py", line 67, in __init__
resolver = RefResolver.from_schema(schema)
File "/home/axel/src/spack/lib/spack/external/jsonschema/validators.py", line 260, in from_schema
return cls(schema.get(u"id", u""), schema, *args, **kwargs)
File "/home/axel/src/spack/lib/spack/external/jsonschema/validators.py", line 245, in __init__
for id, validator in iteritems(meta_schemas)
File "/home/axel/src/spack/lib/spack/external/jsonschema/_utils.py", line 20, in __init__
self.store.update(*args, **kwargs)
File "/home/axel/src/spack/lib/spack/external/jsonschema/validators.py", line 244, in <genexpr>
(id, validator.META_SCHEMA)
File "/home/axel/miniconda3/lib/python3.6/_collections_abc.py", line 744, in __iter__
yield (key, self._mapping[key])
File "/home/axel/src/spack/lib/spack/external/jsonschema/_utils.py", line 23, in __getitem__
return self.store[self.normalize(uri)]
File "/home/axel/src/spack/lib/spack/external/jsonschema/_utils.py", line 16, in normalize
return urlsplit(uri).geturl()
File "/home/axel/src/spack/lib/spack/external/jsonschema/compat.py", line 37, in urlsplit
scheme, netloc, path, query, fragment = _urlsplit(url)
File "/home/axel/miniconda3/lib/python3.6/urllib/parse.py", line 398, in urlsplit
url, scheme, _coerce_result = _coerce_args(url, scheme)
RecursionError: maximum recursion depth exceeded
Information on your system
Python 3.6.4 from Anaconda ruamel.yaml 0.15.37 Debian 9.5
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 24 (24 by maintainers)
Commits related to this issue
- Spack doesn't run site customization when started + vendor setuptools fixes #9206 fixes #9034 A possible solution to #9206 is to avoid running `import site` and all the initialization procedures tha... — committed to epfl-scitas/spack by alalazo 8 years ago
- Spack doesn't run site customization when started + vendor setuptools fixes #9206 fixes #9034 A possible solution to #9206 is to avoid running `import site` and all the initialization procedures tha... — committed to epfl-scitas/spack by alalazo 8 years ago
- Spack doesn't run site customization when started + vendor setuptools fixes #9206 fixes #9034 A possible solution to #9206 is to avoid running `import site` and all the initialization procedures tha... — committed to epfl-scitas/spack by alalazo 8 years ago
- Spack doesn't run site customization when started fixes #9206 fixes #9034 A possible solution to #9206 is to avoid running `import site` and all the initialization procedures that this entails. This... — committed to epfl-scitas/spack by alalazo 8 years ago
- Spack doesn't run site customization when started fixes #9206 fixes #9034 A possible solution to #9206 is to avoid running `import site` and all the initialization procedures that this entails. This... — committed to epfl-scitas/spack by alalazo 8 years ago
- The vendored ruamel.yaml is imported under the namespace 'external' fixes #9206 ruamel.yaml generates a .pth file when installed via pip that has the effect of always preferring the version of this ... — committed to epfl-scitas/spack by alalazo 8 years ago
- Delete references to ruamel.yaml at Spack start-up, if they are present fixes #9206 ruamel.yaml generates a .pth file when installed via pip that has the effect of always preferring the version of t... — committed to epfl-scitas/spack by alalazo 8 years ago
- bugfix: work around ruamel.yaml vendoring issues (#9725) - Delete references to ruamel.yaml at Spack start-up, if they are present - ruamel.yaml generates a .pth file when installed via pip that h... — committed to spack/spack by alalazo 8 years ago
Just to state explicitly another solution that is similar to @ax3l proposal, but doesn’t require changes in vendored packages: if we had a different directory layout for dependencies, we could have embedded them into a namespace. At that point we would use, for instance, something like:
instead of
in Spack’s code. On the plus side we can refer to our vendored packages unambiguously, on the minus side a typo (
from six import ...instead offrom spack.externals.six import ...) could make us use both the system package and the vendored one in different parts of Spack. As a note: this is howsetuptoolsand other similar packages vendor their dependencies.Found a Docker reproducer! 😃
Python2:
Python3:
😱 I think I got it. Installing
ruamel.yaml(viapipfor instance) creates this nastyruamel.yaml-0.15.66-py2.7-nspkg.pthfile insite_packagesordist-packages:According to the docs (emphasis mine):
The creator of
ruamel.yamlseems to be exploiting the mechanism in bold characters to execute code that importsruamel.yamlfromsite-packages/dist-packagesno matter what.I don’t really understand the purpose of that but found a couple of posts here and here. Also, there’s a comment here that suggests the deletion of the infamous issue 28, where the intentions were clearly explained?
To end this: it’s by design ~and I see no simple way out (rather than hacking our way through
sys.modules).~I just submitted a PR that tweaks
sys.modules(#9725). The downside of that approach is that previous references toruamel.yaml, if present, will still point to thesitemodule - I don’t change the object insys.modules(asimportlib.reloaddoes) but just delete an object from cache and replace it. On the other hand it’s the approach which minimizes the lines that have to be changed in core.@scheibelp @tgamblin (Replying to a comment from Peter in #9261)
I’m half-way through having all vendored packages used under the
external.<pkg>namespace in a local branch. From the work involved I see a few relevant downsides to this approach:externalnamespace) and packages that import part of themselves with absolute imports, rather than relative imports.jinja2is also kind of peculiar, as it generates valid python code that contains imports of the package that probably need to be modified.As a sidenote, while I started with the idea of having everything under the
spack.external.<pkg>namespace that was not possible because we have already 2 namespaces at the same level that make use of the vendored dependencies:llnlandspack. This is probably not the most flexible choice to allow changing the file layout in the future.EDIT: going this route consistently is really cumbersome, but if we accept to use namespace
externaljust forruamel.yamlthe changes are not that many. I’ll submit a PR soon.@ibaned If you read the thread you’ll see that #9261 has been implemented already, and it’s waiting for review. As a temporary solution, you can work-around the issue using a virtualenv without
ruamel.yamlinstalled (in that way Spack will use the one that is vendored).For the record: I have a possible solution in #9261 - I just need to make it work correctly with virtualenv and Python 2.7 to pass the tests in Travis.
@ax3l Changing name to the module would work, if we are fine with this hack. In any case, the entire Python initialization process seems like an hack, so fine with me 😄
@alalazo: do you want to do a hot fix to “undo” the sys.modules modification the
pthfile makes? Seems like that is the only option for us at the moment… I think that and contacting the upstream developers are the best bet…To finish the diagnosis:
works as expected. Probably it’s worth opening an issue upstream saying that the package can’t be vendored because of the way it is installed.
@tgamblin Still digging. Right now I can confirm what you say. The code fails here: https://github.com/spack/spack/blob/ff13f3953e700c92117314d46bc1a338f817bb6d/lib/spack/llnl/util/lang.py#L552
Because of:
If I look at the
ruamel.yamlmodule:I could reproduce on Ubuntu, the trigger seems to be having a site
ruamel.yamlpackage installed.I’m a little puzzled by this. I tried your docker image and I can reproduce the problem, but I don’t know why it’s not working.
ruamelis in Spack’slib/spack/externallocation, and that path is clearly put at the front ofsys.pathin thespackscript.I am seeing something kind of like this, where I can’t seem to get it to import the
ruamelfrom Spack’s vendor directory. I wonder if there are weird settings about the system path on Debian. I’ll keep looking.@alalazo @healther @scheibelp @tgamblin after bisecting this problem, I found it was introduced in 63004e3de1473254c3b2a75f1e06afaabdacd7dd
yaml: use ruamel.yaml instead of pyyamlMy system uses: