spack: list_url not working

One of the packages I was trying to install as a dependency has been updated. There is a list_url in the package file but it does not get checked when fetching.

   homepage = "https://github.com/jeroenooms/jsonlite"                                                                 
    url      = "https://cran.r-project.org/src/contrib/jsonlite_0.9.21.tar.gz"                                          
    list_url = "https://cran.r-project.org/src/contrib/Archive/jsonlite"                                                

    version('0.9.21', '4fc382747f88a79ff0718a0d06bed45d') 
spack fetch r-jsonlite
==> Trying to fetch from file:///home/gjohnson/spack/var/spack/cache/r-jsonlite/r-jsonlite-0.9.21.tar.gz
curl: (37) Couldn't open file /home/gjohnson/spack/var/spack/cache/r-jsonlite/r-jsonlite-0.9.21.tar.gz
==> Fetching from file:///home/gjohnson/spack/var/spack/cache/r-jsonlite/r-jsonlite-0.9.21.tar.gz failed.
==> Trying to fetch from https://cran.r-project.org/src/contrib/jsonlite_0.9.21.tar.gz

curl: (22) The requested URL returned error: 404 Not Found
==> Fetching from https://cran.r-project.org/src/contrib/jsonlite_0.9.21.tar.gz failed.
==> Error: All fetchers failed for r-jsonlite-0.9.21-oegoqbzsaflpwyndb5qs5q7dhwayjxw3

The file exists at the URL specified by list_url. https://cran.r-project.org/src/contrib/Archive/jsonlite/ https://cran.r-project.org/src/contrib/Archive/jsonlite/jsonlite_0.9.21.tar.gz

Any ideas?

Thanks.

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

After moving the initialization of the pool in _spider (pool = Pool(processes=len(subcalls))) out of the try block it resides in, placing it before it, you get:

>>> pages, links = spack.util.web._spider(('https://mesa.freedesktop.org/archive/', set(), 'https://mesa.freedesktop.org/archive/', None, 1, 3, True))
==> Error in _spider: daemonic processes are not allowed to have children
==> Error in _spider: daemonic processes are not allowed to have children
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ketl/spack/lib/spack/spack/util/web.py", line 138, in _spider
    results = pool.map(_spider, subcalls)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
AssertionError: daemonic processes are not allowed to have children

So it would seem recursing within _spider using multiprocessing is not allowed.

That means _spider likely requires some rewriting.

A couple of observations:

  • list_depth likely hasn’t seen much use (or users just haven’t noticed or cared enough to report it earlier)
  • Even if the existing code worked, it could potentially cause a fork bomb (when maxdepth>2, and there are many links on all levels of the hierarchy).

Yea, I can work around it but I wanted to report the issue because I see the potential for a whole bunch of packages breaking when updated upstream.

Thanks.