dask: ⚠️ Upstream CI failed ⚠️

https://github.com/dask/dask/issues/9499 is the remaining upstream failure. That test was temporarily xfailed in https://github.com/dask/dask/pull/9670 in order to get our upstream CI build passing. I’m going to go ahead and close this issue so that when a new upstream test failure pops up, a new issue will be opened (which we’re more likely to see). Thanks all!

Thanks @charlesbluca!

After running through the entire test suite to reproduce, I notice that from importlib import metadata still seems to work, so a short term unblocker could be to use that import style instead in the relevant places.

Hrm, that’s strange but not an unreasonable change if it gets things passing. Would you mind pushing up a PR for this?

Sure I can take a look into this

Thanks for writing this up @ncclementi! Please feel free to pick up either of those PRs, or we can work on them together if you like.

Breakdown of upstream failures:

numeric_only

• Most of the failures are related to the change in numeric_only default behavior - see https://github.com/dask/dask/pull/9241 and https://github.com/dask/dask/pull/9269 for reference

No numeric_only

https://github.com/dask/dask/pull/9271 Julia filter most of the numeric only problems and the remaining issues are

• test_roundtrip_parquet_dask_to_spark

E           TypeError: Cannot cast DatetimeArray to dtype datetime64

• test_map_partitions_with_delayed_collection and test_dropna (breakdown issue https://github.com/dask/dask/issues/9365 and proposed fix https://github.com/dask/dask/pull/9366)

E           ValueError: Metadata inference failed in `dropna`.
E           
E           You have supplied a custom function and Dask is unable to 
E           determine the type of output that that function returns. 
E           
E           To resolve this please provide a meta= keyword.
E           The docstring of the Dask function you ran should have more information.
E           
E           Original error is below:
E           ------------------------
E           TypeError('You cannot set both the how and thresh arguments at the same time.')
E           
E           Traceback:
E           ---------
E             File "/home/runner/work/dask/dask/dask/dataframe/utils.py", line 182, in raise_on_meta_error
E               yield
E             File "/home/runner/work/dask/dask/dask/dataframe/core.py", line 6406, in _emulate
E               return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
E             File "/home/runner/work/dask/dask/dask/utils.py", line 1070, in __call__
E               return getattr(__obj, self.method)(*args, **kwargs)
E             File "/usr/share/miniconda3/envs/test-environment/lib/python3.9/site-packages/pandas/util/_decorators.py", line 314, in wrapper
E               return func(*args, **kwargs)
E             File "/usr/share/miniconda3/envs/test-environment/lib/python3.9/site-packages/pandas/core/frame.py", line 6530, in dropna
E               raise TypeError(

dask/dataframe/utils.py:203: ValueError

• test_append_overlapping_divisions

E   RuntimeError: file metadata is only available after writer close

and

E   pyarrow.lib.ArrowInvalid: Casting from timestamp[ns] to timestamp[us] would lose data: 1640995373146292585

pyarrow/error.pxi:100: ArrowInvalid

• test_parquet[fastparquet]

E       TypeError: 'NoneType' object is not iterable

/usr/share/miniconda3/envs/test-environment/lib/python3.9/site-packages/fastparquet/writer.py:1331: TypeError