pandas: BUG: read_csv returns object dtype for dates in empty frame
Code Sample, a copy-pastable example if possible
>>> import io
>>> import pandas as pd
>>> csv = lambda: io.StringIO('date')
>>> df = pd.read_csv(csv(), dtype=int)
>>> df.dtypes
date int64
dtype: object
>>> df = pd.read_csv(csv(), parse_dates=['date'])
>>> df.dtypes
date object
dtype: object
>>> df.date.dt.tz_localize('UTC')
Traceback (most recent call last):
File "<ipython-input-49-067c7ce27232>", line 1, in <module>
df.date = df.date.dt.tz_localize('UTC')
File "~/anaconda/lib/python3.5/site-packages/pandas/core/generic.py", line 2740, in __getattr__
return object.__getattribute__(self, name)
File "~/anaconda/lib/python3.5/site-packages/pandas/core/base.py", line 241, in __get__
return self.construct_accessor(instance)
File "~/anaconda/lib/python3.5/site-packages/pandas/core/series.py", line 2743, in _make_dt_accessor
raise AttributeError("Can only use .dt accessor with datetimelike "
AttributeError: Can only use .dt accessor with datetimelike values
>>> parser = lambda x: pd.to_datetime(x, utc=True)
>>> df = pd.read_csv(csv(), parse_dates=['date'], date_parser=parser)
>>> df.dtypes
date object
dtype: object
>>> df.date.dt.tz_convert('EST')
Traceback (most recent call last):
File "<ipython-input-129-3d458af8513c>", line 1, in <module>
df.date.dt.tz_convert('EST')
File "~/anaconda/lib/python3.5/site-packages/pandas/core/generic.py", line 2740, in __getattr__
return object.__getattribute__(self, name)
File "~/anaconda/lib/python3.5/site-packages/pandas/core/base.py", line 241, in __get__
return self.construct_accessor(instance)
File "~/anaconda/lib/python3.5/site-packages/pandas/core/series.py", line 2743, in _make_dt_accessor
raise AttributeError("Can only use .dt accessor with datetimelike "
AttributeError: Can only use .dt accessor with datetimelike values
Problem description
When reading CSVs with no data rows, read_csv()
returns the dtype object
for dates, which can raise errors on later manipulation. This is contrary to the general behaviour of read_csv()
, which otherwise correctly sets dtypes for empty frames when those dtypes are explicitly passed.
I don’t think it would be hard to return the correct dtype here? If date_parser
is not set, we know the dtype is datetime64[ns]
; otherwise, we can call the parser with empty data, and use the returned dtype.
Note that e.g. read_csv(..., dtype='datetime64[ns]')
is not a solution, as this throws an error when the csv is non-empty.
Expected Output
date int64
dtype: object
date datetime64[ns]
dtype: object
date datetime64[ns, UTC]
dtype: object
Output of pd.show_versions()
INSTALLED VERSIONS
commit: None python: 3.5.2.final.0 python-bits: 64 OS: Darwin OS-release: 16.4.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: None LOCALE: None.None
pandas: 0.19.2 nose: 1.3.7 pip: 9.0.1 setuptools: 27.2.0 Cython: 0.25.2 numpy: 1.11.3 scipy: 0.18.1 statsmodels: 0.6.1 xarray: 0.9.1 IPython: 4.2.0 sphinx: 1.5.1 patsy: 0.4.1 dateutil: 2.6.0 pytz: 2016.10 blosc: None bottleneck: 1.2.0 tables: 3.3.0 numexpr: 2.6.1 matplotlib: 1.5.3 openpyxl: 2.4.0 xlrd: 1.0.0 xlwt: 1.1.2 xlsxwriter: 0.9.6 lxml: 3.7.2 bs4: 4.5.3 html5lib: None httplib2: None apiclient: None sqlalchemy: 1.1.4 pymysql: None psycopg2: None jinja2: 2.8.1 boto: 2.45.0 pandas_datareader: None
About this issue
- Original URL
- State: open
- Created 7 years ago
- Comments: 24 (23 by maintainers)
Commits related to this issue
- ENH: Handle generic timestamp dtypes with Series We only use the nanosecond frequency, so generic timestamp frequencies should be interpreted with the nanosecond frequency. xref gh-15524 (comment). — committed to forking-repos/pandas by gfyoung 7 years ago
- ENH: Handle generic timestamp dtypes with Series We only use the nanosecond frequency, so generic timestamp frequencies should be interpreted with the nanosecond frequency. xref gh-15524 (comment). — committed to forking-repos/pandas by gfyoung 7 years ago
- ENH: Handle generic timestamp dtypes with Series We only use the nanosecond frequency, so generic timestamp frequencies should be interpreted with the nanosecond frequency. xref gh-15524 (comment). — committed to forking-repos/pandas by gfyoung 7 years ago
- DEPR: Deprecated generic timestamp dtypes We only use the nanosecond frequency, and numpy doesn't even handle generic timestamp dtypes well. xref gh-15524 (comment). — committed to forking-repos/pandas by gfyoung 7 years ago
- DEPR: Deprecate generic timestamp dtypes We only use the nanosecond frequency, and numpy doesn't even handle generic timestamp dtypes well. xref gh-15524 (comment). — committed to forking-repos/pandas by gfyoung 7 years ago
- DEPR: Deprecate generic timestamp dtypes We only use the nanosecond frequency, and numpy doesn't even handle generic timestamp dtypes well. xref gh-15524 (comment). — committed to forking-repos/pandas by gfyoung 7 years ago
- DEPR: Deprecate generic timestamp dtypes (#15987) * DEPR: Deprecate generic timestamp dtypes We only use the nanosecond frequency, and numpy doesn't even handle generic timestamp dtypes well. ... — committed to pandas-dev/pandas by gfyoung 7 years ago
Well, if I call
read_csv
with a specific column inparse_dates
, I definitely want the column to be a datetime, and since the column is empty, it will unequivocally succeed in that conversion – it’s not like it can contain a parse error.I’m not sure I follow you on the second paragraph? What won’t work in my example?
Concatting an empty frame of dtype
object
to a non-empty one of dtypedatetime64[ns, UTC]
throws an error, is that not intended?Anyway, potential use-cases are (a) concatting together multiple csvs, some of which may be empty; (b) regularly loading a csv which is frequently updated, and may or may not be empty. In (a), I would have to assign timezones after concat to stop it throwing an error; in (b), I would have to specifically check whether the frame is empty before attempting any timezone manipulation.
The workarounds aren’t too hard, it just seems a bit awkward when the simple version would work fine for any other dtype.