pandas-stubs: `concat` of types `Iterable[Any]` cause Pyright to think subsequent code is unreachable because of recent stubs change

Describe the bug In the following code the last line is wrongly considered to be unreachable by Pyright:

from typing import Any

import pandas as pd


def generate_series() -> Any:
    return pd.Series([1, 2, 3])


df = pd.concat({k: generate_series() for k in range(10)}, axis=1)

print("Unreachable?!")

This is caused by the recent change https://github.com/pandas-dev/pandas-stubs/pull/858 adding a stub returning Never. This is presumably a misuse of Never or the stub as a whole should be moved to the very end so that concat on types Iterable[Any] can first match a stub with valid return.

To Reproduce See https://github.com/microsoft/pylance-release/issues/5631 and https://github.com/microsoft/pylance-release/issues/5630 for issues reported on the Pylance repo.

Please complete the following information:

OS: Linux
OS Version 5.15.0-46-generic
Python 3.11.6
Pylance v2024.3.1
version of installed pandas-stubs: the one shipped with above Pylance version

About this issue

Original URL
State: closed
Created 4 months ago
Reactions: 7
Comments: 38 (5 by maintainers)

Most upvoted comments

Alright, I give up. I’m thoroughly convinced at this point that I’ve made a clear and strong argument here that you guys are far from “trying to find something that is ‘just right’” and rather obsessing over a narrow corner-case in a way that provably isn’t helping anyone in this particular instance. Rationalize it all you will, fact is people are having issues because of your changes and are hacking the stubs locally to circumvent them.

Also the fact remains that logically what you did is plain wrong: it’s not true that concat never returns anything on Iterable[Any]input, and, because of your changes, Pyright/the stubs do instead report that. All the discussion about how then Pylance interprets that report from Pyright/the stubs is then a red herring IMO. If anything you could start an argument that Pyright’s method of returning the first matching stub is limited, but I’m not participating.

+12

torext on Mar 14, 2024

For those following this thread, I found a fix in #890 that supports our use case (disallow Iterable[None]) and also avoid the “unreachable code” aspect if the type checker infers the argument to be list[Any]

Dr-Irv on Mar 15, 2024

Yes, that’s true, but the question is whether our goal is to support well-typed code that calls pandas, or untyped code that calls pandas. We’ve chosen the former from a philosophical point of view.

Apologies for my ignorance, but shouldn’t untyped code be considered the default case? Unless these stubs are widely used by different IDEs and type checkers then what is the point of making them? And unless the latter don’t behave sanely and usefuly for the base-case of untyped code, then what’s the point of those in turn?

My point being, again, it seems to me that you are prioritising here a very particular edge case purely for the sake of prioritising that niche case. I on the other hand have work to do, and I need a solid cross-OS IDE for various languages and in particular Python.

Apologies for the likely palpable frustration on my side, but I hope I’m making it clear where I’m coming from on this. You said you guys made a philosophical choice, I’m instead urging you to consider pragmatism.

torext on Mar 13, 2024

The issue you created (https://github.com/microsoft/pylance-release/issues/5640#issuecomment-1997327840) doesn’t address the bug I’m reporting here! I’m reporting here an issue where concaton a type Iterable[Any]returns a stub with return type Never. Clearly even the most hard-core type-enthusiast can see this is wrong: it’s simply not true that such a call will never return. It only sometimes won’t return (in fact only in the very rare case that all objects in the iterable are None).

I don’t have type checking set to off in Pylance and I don’t plan to. I want some basic type checking and stubs that do the right thing, even if there’s some Anys in my code. You seriously cannot hope to catch all bugs and errors with static type checking; if that was the goal, then Python as a whole should’ve been designed differently (e.g. statically typed).

torext on Mar 14, 2024

So, is it an issue in PyLance default configuration?

It’s an issue in the default configurations of both Pylance and Pyright (when run as a language server).

debonte on Mar 14, 2024

I think https://github.com/pandas-dev/pandas/issues/57846 would be the ideal solution as it would avoid the false-negatives and false-positives mentioned in this issue: the Never overload would no longer be needed and Nones are no longer accepted.

As long as there is no pushback at pandas, I wouldn’t mind making the change here at pandas-stubs before pandas releases a new version that has this change

twoertwein on Mar 14, 2024

It’s unclear to me whether we should be supporting the use of parameters that are typed as Any by users and passed into methods declared in the stubs, because that defeats the purpose of type checking, IMHO.

This scenario may be more interesting than explicit Any:

pd.concat([])
print("Unreachable?!")

FWIW, Pylance had two issues filed on this (one with 5 up votes) within about 12 hours of shipping this change.

debonte on Mar 13, 2024