pandas: BUG: inconsistant parsing between Timestamp and to_datetime

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

pd.Timestamp('10 june 2000 8:30')
>>>
Timestamp('2000-06-10 08:30:00')

pd.to_datetime('10 june 2000 8:30')
>>>
 UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.
  pd.to_datetime('10 june 2000 8:30')

Timestamp('2000-06-10 08:30:00')

Issue Description

to_datetime used to be able to parse date using month in English. There is no ambiguity in this case (no doubt what is the day position), but it raises now a UserWarning.

Surprisingly, a Timestamp can be constructed without any warning with the same string that raised a Warning with to_datetime

Expected Behavior

Ideally, I would like to have no user warning with a string such as ‘8 July 2010’ or ‘July 8 2012’. At the minimum to_datetime and Timestamp should raise on the same strings.

Installed Versions

INSTALLED VERSIONS

commit : c2a7f1ae753737e589617ebaaff673070036d653 python : 3.11.0.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19044 machine : AMD64 processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : fr_FR.cp1252

pandas : 2.0.0rc1 numpy : 1.23.5 pytz : 2022.7 dateutil : 2.8.2 setuptools : 65.6.3 pip : 23.0.1 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.2 html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.10.0 pandas_datareader: None bs4 : 4.11.1 bottleneck : 1.3.5 brotli : fastparquet : None fsspec : None gcsfs : None matplotlib : 3.7.1 numba : None numexpr : 2.8.4 odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : None snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None zstandard : None tzdata : None qtpy : None pyqt5 : None

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (15 by maintainers)

Most upvoted comments

You are right.

pd.read_csv('data.csv',` index_col='Date', parse_dates=True, date_format="%m/%d/%Y %I:%M:%S `%p")

parses correctly

ax-va on Jul 18, 2023

your format’s not correct, you’re missing the meridiem directive

MarcoGorelli on Jul 17, 2023

@MarcoGorelli you were faster than me!

Apologies - I’d normally prefer to mentor people through contributions, but with the final 2.0.0 release on the horizon (possibly next week!), I figured I’d just go ahead and try to get this in

Regarding the warning - I’ll try to explain what’s going on:

first, pandas tries to infer the format. It can’t guess all formats, but if it can guess the format of the given element, then it will be parsed using pandas’ own parsers
if pandas can’t infer the format, then a library called dateutil (which is one of pandas’ dependencies) will be used to parse the elements one-by-one. This is slower, but also riskier because dateutil doesn’t tell us which format it used, and so there’s no guarantee that the same format will be used for each element

So, this is what the warning is trying to tell users about. If you have suggestions for how to better phrase it, that would be extremely welcome

is there any documentation on how to seamlessly navigate python and cython/C code in an IDE

I wish 😄 But the legendary @WillAyd has some blog posts on debugging C extensions at https://willayd.com/

MarcoGorelli on Mar 25, 2023

thanks for the report

this is expected: to_datetime can work on an array of elements, so it needs to check that they can all be parsed with the same format, whereas Timestamp only parses a single element so there’s no risk of inconsistency between elements

having said that, the warning could probably be extended to say

Could not infer format, so each element will be parsed individually, falling back to `dateutil`.
To ensure parsing is consistent and as-expected, please specify a format.
Or, to accept the current behavior, pass `format='mixed'`

MarcoGorelli on Mar 24, 2023