fastparquet: ParquetFile error not finding _metadata directory

system: centos 7.3, fastparquet 0.1.2, anaconda python 3.6.2

I created some parquet files using

fastparquet.write(filename,data,compression='SNAPPY')

and now when I try to read those same files with

pf=ParquetFile(filename)

I get the error:

NotADirectoryError: [Errno 20] Not a directory: 'filename/_metadata'

thoughts?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (7 by maintainers)

Most upvoted comments

Hello, I have the same problem, but I didn’t understand your solution above. Can you elaborate? Thank you,my fastparquet version is 0.1.5

Wercurial on Jan 7, 2020

# ls -al
total 16961
drwxr-xr-x 2 root root     4096 Oct 16 23:34 .
drwxr-xr-x 3 root root     4096 Oct 16 23:34 ..
-rw-r--r-- 1 root root 17359516 Oct 16 23:34 file1.parq

# python
Python 3.6.2 |Anaconda, Inc.| (default, Sep 30 2017, 18:42:57)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastparquet
>>> fastparquet.ParquetFile('file1.parq')
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 96, in __init__
    with open_with(fn2, 'rb') as f:
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/util.py", line 44, in default_open
    return open(f, mode)
NotADirectoryError: [Errno 20] Not a directory: 'file1.parq/_metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 102, in __init__
    self._parse_header(f, verify)
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 125, in _parse_header
    self._set_attrs()
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/api.py", line 139, in _set_attrs
    self.schema = schema.SchemaHelper(self._schema)
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/schema.py", line 80, in __init__
    self.text = schema_to_text(self.schema_elements[0])
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/schema.py", line 47, in schema_to_text
    text += '\n' + schema_to_text(child, indent)
  File "/opt/anaconda3/lib/python3.6/site-packages/fastparquet/schema.py", line 36, in schema_to_text
    root.converted_type])
KeyError: 24

virtualluke on Oct 16, 2017