ibis: bug: Incomplete type hints

What happened?

While evaluating Ibis at work, I found some unexpected mypy errors:

import ibis
import ibis.selectors as s
from ibis import deferred as d_

table = ibis.table({"col1": "int"})

# error: Argument 1 to "order_by" of "Table" has incompatible type "Value"; expected "str | Column | tuple[str | Column, bool] | Sequence[str] | Sequence[Column] | Sequence[tuple[str | Column, bool]] | None"  [arg-type]
table.order_by(ibis.desc("col1"))

# error: Argument 1 to "mutate" of "Table" has incompatible type "Across"; expected "Sequence[Expr] | None"  [arg-type]
table.mutate(s.across(["col1"], d_.mean()))

# error: Argument "col1_again" to "mutate" of "Table" has incompatible type "Deferred"; expected "Value"  [arg-type]
table.mutate(col1_again=d_["col1"])

I’ll keep adding to this issue in case I’d find more. Also, a big thank you for making Ibis a typed library! 😃

What version of ibis are you using?

Ibis 6.1 Mypy 1.4.0 Python 3.11

What backend(s) are you using, if any?

No response

Relevant log output

No response

Code of Conduct

I agree to follow this project’s Code of Conduct

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (17 by maintainers)

Most upvoted comments

The vast majority of ibis was implemented before type hints became widespread and it’s a lot of work to add them.

I can relate to that. It’s the same story for Vega-Altair where I’m currently in the process of typing at least the public API and it’s not easy…

I know @jcrist has thoughts here. I think there may be a way we can ensure that our type hints are as-expected without having to annotate the entire codebase.

I think the issue title I chose is misleading. Mypy allows for gradually typing existing codebases and so it’s fine if Ibis is not completely typed. Issues arise when the provided type hints are either wrong (e.g. specified types on function input arguments cannot be processed by function) or incomplete (e.g. a union of types does not cover all accepted types) where the later one seems to be the case for the 3 errors I mentioned above.

The process which we follow for Altair and what I’ve seen with other libraries is that

mypy is implemented first and all existing errors are fixed
more type hints are gradually introduced starting with the parts of the library that benefit from it the most
once you’re happy with the coverage (doesn’t have to be 100%) you add the py.typed file to tell type checkers such as mypy that the type hints can be used for type checking in other code bases.

By introducing the py.typed file in #5304 without having mypy enabled, these errors now show up in other code bases but not yours and users will have to add # type: ignore comments to get mypy to pass.

Based on #2823 and the related PRs, I take it that it’s not easy to fix all existing mypy issues. For what it’s worth, I think you could either remove again the py.typed file so that other code bases are not affected, or introduce mypy for checking and get it to pass. You could remove type hints in the parts of the code which are tricky (or use typing.Any) so that mypy does not check them.

Running mypy ibis --ignore-missing-imports after pip install -e . gives me 130 individual errors but many of them have the same root causes.

binste on Aug 12, 2023

Hopefully this isn’t a blocker for you, but let us know!

cpcloud on Aug 12, 2023