griffe: GriffeLoader breaks dealing with hidden .venv folder

Describe the bug

During my CI a .venv folder is created locally to run tests. The presence of this .venv folder then breaks the docs generation.

When I attempt to run mkdocs build when the .venv folder is present, I get the following error message:

ERROR    -  Error reading page '<redacted filepath>/<redacted filename>.md': ''
Traceback (most recent call last):
  File "/Users/rsenseman/Development/<redacted project name>/docs/.venv/lib/python3.8/site-packages/griffe/loader.py", line 550, in _get_or_create_parent_module
    parent_module = parent_module[parent_part]
  File "/Users/rsenseman/Development/<redacted project name>/docs/.venv/lib/python3.8/site-packages/griffe/mixins.py", line 31, in __getitem__
    return self.members[parts[0]][parts[1:]]  # type: ignore[attr-defined]
KeyError: ''

This is unintuitive because the error message makes it look like there is some issue with the legitimate file <redacted filename>.md, but in reality what is breaking is some code in the GriffeLoader class that as far as I can tell is attempting to built out the directory structure of the repo.

Here’s what the file structure of the Python project looks like with this .venv folder: Screenshot 2023-07-24 at 4 35 41 PM

I’m not exactly sure why the .env folder specifically causes an issue, but regardless hidden/dot folder should just be ignored. I tried to use the mkdocs-exclude plugin to see if the exclusions would work with mkdocstrings but that was unsuccessful.

Debug logs Relevant logs with sensitive project information excluded:

ERROR    -  Error reading page '<redacted filepath>/<redacted filename>.md': ''
Traceback (most recent call last):
  File "/Users/rsenseman/Development/<redacted project name>/docs/.venv/lib/python3.8/site-packages/griffe/loader.py", line 550, in _get_or_create_parent_module
    parent_module = parent_module[parent_part]
  File "/Users/rsenseman/Development/<redacted repo name>/docs/.venv/lib/python3.8/site-packages/griffe/mixins.py", line 31, in __getitem__
    return self.members[parts[0]][parts[1:]]  # type: ignore[attr-defined]
KeyError: ''

To Reproduce Steps to reproduce the behavior:

Have virtual environment .venv folder in Python project files.
Run mkdocs build
This will raise the error

Expected behavior

I would expect all contents of hidden repos to be ignored.

System (please complete the following information):

griffe version: 0.32.3
Python version: 3.8.13
OS: mac

Additional context

I can implement a fix for this after opening the issue

About this issue

Original URL
State: closed
Created a year ago
Comments: 19 (10 by maintainers)

Commits related to this issue

bug [finder]: ignore hidden folders When searching module subpaths to load files for parsing, skip hidden folders that start with the `.` character. Issue #185: https://github.com/mkdocstrings/griff... — committed to rsenseman/griffe by rsenseman a year ago
fix: Skip early submodules with dots in their path Issue #185: https://github.com/mkdocstrings/griffe/issues/185 — committed to mkdocstrings/griffe by pawamoy 10 months ago

Most upvoted comments

Fair enough. Thank you for considering the change and providing a thoughtful response to the pros and cons in addition to identifying the actual root cause of the issue.

rsenseman on Jul 28, 2023

It’s true that I have answered your questions with other questions a few times. Sorry about that.

Now I wouldn’t say I am nitpicking. I can understand that from your side, you focus on treating the symptom, i.e. modifying the code so that your use-case is better supported. You should however understand that I’m more interested in treating the cause. It may seem like ignoring dot-files/directories is an improvement, but doing that would actually validate a use-case that I consider unsupported. Accepting this change is opening the door to more requests for ignoring other files and folders, but this is not how Griffe was designed. Then later, I’ll get issues telling me that using Griffe that way does not generate a correct inventory in mkdocstrings and prevents cross-referencing objects. Maybe you don’t care about that today, but you probably will in the future, or someone else will. You probably also know that blindly accepting changes (even one-liners) from contributors who don’t know much about the project is generally not a good idea. There’s a maintenance cost to everything. I’ll be the one paying it. So yes, I think back and forth is needed, and this is generally the process that is followed in open-source projects. We discuss about an issue before designing the necessary changes and sending pull requests. So my answer to your previous question “do you think ignoring hidden folders is a good idea” is, no, I don’t think so. That would be option 3.

You do make a point about improving the error message though, or improving handling of the error, and that’s what I will do: I’ll probably catch the key error that is triggered by trying to get a member with an empty name, coming from dot-splitting the member name (.venv -> ['', 'venv']). The rest of the code will already ignore all found modules inside this folder because it doesn’t have a top-level __init__.py module.

pawamoy on Jul 28, 2023