griffe: Unable to build documentation for the ibis project

Describe the bug A clear and concise description of what the bug is.

When running mkdocs build on master of ibis I get the following traceback:

Traceback (most recent call last):
  File "/nix/store/xybc4xyjrwda96sq975x5zsn3szjj46v-python3.10-mkdocs-1.3.0/bin/.mkdocs-wrapped", line 9, in <module>
    sys.exit(cli())
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocs/__main__.py", line 192, in build_command
    build.build(config.load_config(**kwargs), dirty=not clean)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocs/commands/build.py", line 292, in build
    _populate_page(file.page, config, files, dirty)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocs/commands/build.py", line 174, in _populate_page
    page.render(config, files)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocs/structure/pages.py", line 175, in render
    self.content = md.convert(self.markdown)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/markdown/core.py", line 264, in convert
    root = self.parser.parseDocument(self.lines).getroot()
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/markdown/blockparser.py", line 90, in parseDocument
    self.parseChunk(self.root, '\n'.join(lines))
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/markdown/blockparser.py", line 105, in parseChunk
    self.parseBlocks(parent, text.split('\n\n'))
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/markdown/blockparser.py", line 123, in parseBlocks
    if processor.run(parent, blocks) is not False:
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocstrings/extension.py", line 121, in run
    html, handler, data = self._process_block(identifier, block, heading_level)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocstrings/extension.py", line 195, in _process_block
    data: CollectorItem = handler.collect(identifier, options)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/mkdocstrings_handlers/python/handler.py", line 195, in collect
    unresolved, iterations = loader.resolve_aliases(only_exported=True, only_known_modules=True)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/griffe/loader.py", line 179, in resolve_aliases
    self.expand_wildcards(wildcards_module)
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/griffe/loader.py", line 254, in expand_wildcards
    self.expand_wildcards(member, only_known_modules, seen)  # type: ignore[arg-type]
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/griffe/loader.py", line 254, in expand_wildcards
    self.expand_wildcards(member, only_known_modules, seen)  # type: ignore[arg-type]
  File "/nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/griffe/loader.py", line 260, in expand_wildcards
    if new_member.name not in obj.members or obj[new_member.name].lineno < alias_lineno:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

To Reproduce

  1. Set up an ibis development environment (any of nix, conda or pip should reproduce)
  2. Run mkdocs build

Expected behavior

I would expect the docs to build without error šŸ˜ƒ

System (please complete the following information):

  • griffe version:
āÆ poetry show griffe
name         : griffe
version      : 0.19.3
  • Python version: 3.10
  • OS: NixOS (Linux)

Additional context

I dug around a bit with ipython --pdb -m mkdocs build.

It looks like griffe maybe isnā€™t correctly tracking __all__ when itā€™s computed instead of a constant hand-curated list which makes star imports (which we use sparingly in ibis) include aliases from other modules.

Hereā€™s the final stack frame in the traceback of that debugging session:

> /nix/store/4spdlhxmri6h2492n7iar1rk9l903pbw-python3-3.10.4-env/lib/python3.10/site-packages/griffe/loader.py(260)expand_wildcards()
    258
    259         for new_member, alias_lineno, alias_endlineno in expanded:
--> 260             if new_member.name not in obj.members or obj[new_member.name].lineno < alias_lineno:
    261                 obj[new_member.name] = Alias(
    262                     new_member.name, new_member, lineno=alias_lineno, endlineno=alias_endlineno

ipdb> new_member
<Alias('com', 'ibis.common.exceptions')>
ipdb> obj
<Module(PosixPath('ibis/expr/types/__init__.py'))>
ipdb> obj[new_member.name]
<Alias('com', 'ibis.expr.types.analytic.com')>
ipdb> obj[new_member.name].lineno is None
True
ipdb> pp obj.members
{'Analytic': <Alias('Analytic', 'ibis.expr.types.analytic.Analytic')>,
 'Column': <Alias('Column', 'ibis.expr.types.arrays.Column')>,
 'Exists': <Alias('Exists', 'ibis.expr.types.analytic.Exists')>,
 'Expr': <Alias('Expr', 'ibis.expr.types.analytic.Expr')>,
 'Iterable': <Alias('Iterable', 'ibis.expr.types.arrays.Iterable')>,
 'Scalar': <Alias('Scalar', 'ibis.expr.types.arrays.Scalar')>,
 'TYPE_CHECKING': <Alias('TYPE_CHECKING', 'ibis.expr.types.arrays.TYPE_CHECKING')>,
 'TopK': <Alias('TopK', 'ibis.expr.types.analytic.TopK')>,
 'V': <Alias('V', 'ibis.expr.types.arrays.V')>,
 'Value': <Alias('Value', 'ibis.expr.types.arrays.Value')>,
 'analytic': <Module(PosixPath('ibis/expr/types/analytic.py'))>,
 'annotations': <Alias('annotations', 'ibis.expr.types.arrays.annotations')>,
 'arrays': <Module(PosixPath('ibis/expr/types/arrays.py'))>,
 'binary': <Module(PosixPath('ibis/expr/types/binary.py'))>,
 'category': <Module(PosixPath('ibis/expr/types/category.py'))>,
 'collections': <Module(PosixPath('ibis/expr/types/collections.py'))>,
 'com': <Alias('com', 'ibis.expr.types.analytic.com')>,
 'core': <Module(PosixPath('ibis/expr/types/core.py'))>,
 'deprecated': <Alias('deprecated', 'ibis.expr.types.analytic.deprecated')>,
 'generic': <Module(PosixPath('ibis/expr/types/generic.py'))>,
 'geospatial': <Module(PosixPath('ibis/expr/types/geospatial.py'))>,
 'groupby': <Module(PosixPath('ibis/expr/types/groupby.py'))>,
 'inet': <Module(PosixPath('ibis/expr/types/inet.py'))>,
 'json': <Module(PosixPath('ibis/expr/types/json.py'))>,
 'literal': <Alias('literal', 'ibis.expr.types.arrays.literal')>,
 'logical': <Module(PosixPath('ibis/expr/types/logical.py'))>,
 'maps': <Module(PosixPath('ibis/expr/types/maps.py'))>,
 'numeric': <Module(PosixPath('ibis/expr/types/numeric.py'))>,
 'public': <Alias('public', 'ibis.expr.types.analytic.public')>,
 'relations': <Module(PosixPath('ibis/expr/types/relations.py'))>,
 'sortkeys': <Module(PosixPath('ibis/expr/types/sortkeys.py'))>,
 'strings': <Module(PosixPath('ibis/expr/types/strings.py'))>,
 'structs': <Module(PosixPath('ibis/expr/types/structs.py'))>,
 'temporal': <Module(PosixPath('ibis/expr/types/temporal.py'))>,
 'typing': <Module(PosixPath('ibis/expr/types/typing.py'))>,
 'uuid': <Module(PosixPath('ibis/expr/types/uuid.py'))>}

Here com is technically a member of ibis.expr.types.analytic but itā€™s not part of __all__ (computed using @public), so ideally it isnā€™t picked up.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Griffe 0.25.3 and mkdocstrings-python 0.8.3 released šŸš€

+2
pawamoy on Jan 4, 2023

Thank you for all that you do!

+1
cpcloud on Jan 5, 2023

Iā€™ll change the level to DEBUG šŸ‘

+1
pawamoy on Jan 4, 2023

Also, the bug was triggered thanks to the missing type here: https://github.com/ibis-project/ibis/blob/master/ibis/config.py#L110 (interactive, in Attributes section of Repr).

+1
pawamoy on Jan 4, 2023

Created some gists, because the logs are huge:

  1. Issue 2
  2. Issue 3
+1
cpcloud on Jan 4, 2023

Hey @pawamoy, really warms my heart to get this message in my inbox. Iā€™ll give it go and report back!

+1
cpcloud on Jan 4, 2023

Hello @cpcloud, Iā€™ve released quite a number of bugfixes, could you give Griffe another try? Iā€™m having trouble setting up a dev environment for your ibis project, Iā€™ve followed the docs, but am still missing dependencies.when running mkdocs build: ModuleNotFoundError: No module named 'google'.

+1
pawamoy on Dec 23, 2022

So the TypeError: '<' not supported between instances of 'NoneType' and 'int' should be fixed now, in 0.20.0, but youā€™ll still have the cyclic alias errors. Iā€™ll leave this open until itā€™s fixed šŸ™‚

+1
pawamoy on Jun 3, 2022

OK, your wildcard imports are a bit too wild for Griffe.

I was able to fix the issue with line numbers, and then faced multiple cyclic aliases errors. Your code works because Python, but doing inference is hard.

In ibis/backends/dask/execution/join.py:

-from ibis.backends.dask.execution import constants
+from ibis.backends.pandas.execution import constants

Importing from dask works, because there constants are wildcard imported from another dask submodule which imports it from the pandas backends. Actually importing it from pandas reduces the indirections and helps Griffe. Griffe can probably be improved here, with a smarter/more capable wildcard expanding algorithm.

Then more cyclic alias errors that I was able to suppress in different parts of the code.

Serialization still crashes with infinite recursion šŸ˜• But you shouldnā€™t need serialization for building your docs.

Now I have to admit that Griffeā€™s data model is not perfect. It still considers submodules like members, when clearly it should not. In Python it seems that even if you define a member in a module using the same name as one of the moduleā€™s submodules, you can still import from that submodule (import machinery and attribute access are different).

Iā€™ll have to refactor it. It promises to be a challenging task šŸ˜…

Iā€™ll post an update here when I feel like I can push a fix.

+1
pawamoy on May 31, 2022

Thanks for the very detailed report!

Griffe does not support the public package indeed, which means it wonā€™t be able to infer __all__ correctly. That kind of support can be brought through an extension (and I think it would be easy to write: upon visiting classes or functions, if they are decorated with public or public.add, add them to objects exported by the module).

Now the issue here is more that the existing alias with name com in the ibis.expr.types module does not have a line number, when it should. Iā€™ll run my own debugging session to see if I forgot to set this attribute somewhere in the code šŸ™‚

+1
pawamoy on May 29, 2022
Read more comments on GitHub
← griffe: Include the location of the mistake in case of a "Could not resolve X" error
mkdocstrings: mkdocstrings might not work with editable installs in the new setuptools →