pandas: BUG: Pyarrow dependency warning starts with newline which makes it impossible to filter out by message with -W or PYTHONWARNINGS

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Issue Description

I would expect that the Pyarrow warning is easily ignorable by message i.e. something like the following does not produce any warning:

python -W 'ignore:Pyarrow:DeprecationWarning' -c 'import pandas'

The issue is that because the warning starts with a newline it can not be targetted by message. This seems to be the case of any warning that starts with some kind of whitespace character.

I tried a variety of things like -W 'ignore:\nPyarrow:DeprecationWarning' but I could not make it work. I think the reason is the “ignoring any whitespace at the start or end of message” from the warnings documentation:

In -W and PYTHONWARNINGS, message is a literal string that the start of the warning message must contain (case-insensitively), ignoring any whitespace at the start or end of message.

In scikit-learn tests warnings are turned into errors with pytest -W and we would like to use -W too to ignore the Pyarrow dependency warning for now. A possible work-around is to use filterwarnings which accept a regex, see Note in https://docs.pytest.org/en/stable/how-to/capture-warnings.html#controlling-warnings for more details, but being able to use -W or PYTHONWARNINGS seems desirable e.g. to control warnings from the command-line outside of pytest.

Expected Behavior

There is an easy way with -W or PYTHONWARNINGS to ignore the Pyarrow DeprecationWarning e.g.

python -W 'ignore:Pyarrow:DeprecationWarning' -c 'import pandas'

Installed Versions

INSTALLED VERSIONS ------------------ commit : f538741432edf55c6b9fb5d0d496d2dd1d7c2457 python : 3.11.4.final.0 python-bits : 64 OS : Linux OS-release : 6.7.0-arch3-1 Version : #1 SMP PREEMPT_DYNAMIC Sat, 13 Jan 2024 14:37:14 +0000 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.2.0 numpy : 1.26.3 pytz : 2023.3 dateutil : 2.8.2 setuptools : 68.0.0 pip : 23.2 Cython : 0.29.35 pytest : 7.4.0 hypothesis : None sphinx : 6.0.0 blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.18.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : None gcsfs : None matplotlib : 3.8.0 numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.12.0 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None zstandard : 0.21.0 tzdata : 2023.3 qtpy : None pyqt5 : None

About this issue

  • Original URL
  • State: closed
  • Created 5 months ago
  • Comments: 16 (11 by maintainers)

Most upvoted comments

Preemptively +100 to remove the deprecation warning in 2.2.1

@mimizone you probably know this already but you don’t have to use the full message, it only needs to match at the beginning. Since “https://” is towards the end of the message, I don’t think it matters.

For example if there was not the newline at the start you could ignore the message like this:

PYTHONWARNINGS='ignore:Pyarrow:DeprecationWarning'

or

PYTHONWARNINGS='ignore:Pyarrow will become a required dependency:DeprecationWarning'