pandahouse: KeyError: 'Nullable(String)'

I get the following error with one column, even thoI do have other empty columns which do notcause any errors (such as aircraftCategory is a string and is NaN)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-19-d531a02e1a3b> in <module>()
      2 
      3 connection = {'host': 'http://localhost:8123'}
----> 4 df = ph.read_clickhouse('SELECT hex, flight, recorded_date, recorded_datetime,recorded_longitude, recorded_latitude, recorded_squawk, recorded_altitude, recorded_verticalRate,recorded_track,recorded_speed,aircraftCategory FROM flightLog_MT LIMIT 1000000', connection=connection)

~/.pyenv/versions/3.6.2/envs/general_362/lib/python3.6/site-packages/pandahouse/core.py in read_clickhouse(query, tables, index, connection, **kwargs)
     56     lines = execute(query, external=external, stream=True,
     57                     connection=connection)
---> 58     return to_dataframe(lines, **kwargs)
     59 
     60 

~/.pyenv/versions/3.6.2/envs/general_362/lib/python3.6/site-packages/pandahouse/convert.py in to_dataframe(lines, **kwargs)
     60     dtypes, parse_dates, converters = {}, [], {}
     61     for name, chtype in zip(names, types):
---> 62         dtype = CH2PD[chtype]
     63         if dtype == 'object':
     64             converters[name] = decode_escapes

KeyError: 'Nullable(String)'

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 3
  • Comments: 15 (5 by maintainers)

Most upvoted comments

2023 year! Error not fix

File “C:\gmexpert\venv\lib\site-packages\pandahouse\core.py”, line 58, in read_clickhouse return to_dataframe(lines, **kwargs) File “C:\gmexpert\venv\lib\site-packages\pandahouse\convert.py”, line 67, in to_dataframe dtype = CH2PD[chtype] KeyError: ‘Nullable(String)’

@inkrement I’d suggest to use clickhouse-driver or https://github.com/ibis-project/ibis @haakonvt Sadly I don’t have time to implement it, however Ibis has support for it. I highly recommend to try ibis until We have a native Apache Arrow database interface for clickhouse.