filesystem_spec: AttributeError: 'HadoopFileSystem' object has no attribute 'close'

When running the following code:

import fsspec
f = fsspec.open('hdfs:///path/to/file.csv').open()
...

I get the following error when the script exits (but complete successfully):

AttributeError: 'HadoopFileSystem' object has no attribute 'close'
Exception ignored in: 'pyarrow.lib.HadoopFileSystem.__dealloc__'
AttributeError: 'HadoopFileSystem' object has no attribute 'close'

I’m also getting the error when running:

import fsspec
f = fsspec.open('hdfs:///path/to/file.csv').open()
...
f.close()

The versions in my environment are:

python: 3.7.8
fsspec: 0.8.0
pyarrow: 1.0.1

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 24 (24 by maintainers)

Most upvoted comments

Well I suppose that close can be added to the list of known attributes here; but interesting that this was not apparently needed before. Note that this is a method for the class instance, not the file, to close and free the connection, which is happening during garbage collection.