readthedocs.org: Custom LFS integration has broken unexpectedly

@RuRo we have deployed the fix, git-lfs should be working now. Thanks for helping us debugging this!

@RuRo we actually just tracked this down to https://github.com/readthedocs/readthedocs.org/blob/babbdf058f1ae4eade27b4ef579e6b142a5dffba/readthedocs/vcs_support/base.py#L76-L76, the enviroment vars are being updated on the same instance 😃 we are discussing a fix, but your PR should be fine to merge anyway.

Ah, okay. I think, it makes sense now. The #8263 must be the cause after all. Although it is exporting the correct GIT_DIR environment variable, it results in git getting confused about the location of the current worktree.

@humitos can we get a temporary quick fix for this regression, that doesn’t push the GIT_DIR to the user environment?

Edit: actually, the correct solution would probably be to just export GIT_WORK_TREE alongside GIT_DIR in readthedocs/vcs_support/backends/git.py::Backend.env

Here’s a relevant extract from man git-config (core.worktree)

If --git-dir or GIT_DIR is specified but none of --work-tree, GIT_WORK_TREE
and core.worktree is specified, the current working directory is regarded as the
top level of your working tree.

Edit edit: I made a PR with the proposed fix.

Unfortunately, it doesn’t seem, that GIT_DIR is the culprit here. By inspecting the output of env

GIT_DIR=/home/docs/checkouts/readthedocs.org/user_builds/<projectname>/checkouts/latest/.git

and git rev-parse --git-dir we can see, that the .git directory is located in

/home/docs/checkouts/readthedocs.org/user_builds/<projectname>/checkouts/latest/.git

as expected. It’s only the current git worktree root, that is for some reason moved to

/home/docs/checkouts/readthedocs.org/user_builds/<projectname>/checkouts/latest/docs

---

The weirdest part is that according to git worktree --list, the only existing worktree is in

/home/docs/checkouts/readthedocs.org/user_builds/<projectname>/checkouts/latest  <sha> (detached HEAD)

So at first glance it seems like the output of git rev-parse --show-toplevel contradicts the output of git worktree --list.

---

Okay. This is insane. I think, that I might have found the problem. And it might be a bug in git itself. Consider the following example

# Setup example repo
mkdir /example
cd /example
git init .
mkdir docs
cd docs

Then the following behaviour is clearly broken

git rev-parse --show-toplevel # prints /example as expected
export GIT_DIR=/example/.git
git rev-parse --show-toplevel # prints /example/docs for some reason

In fact, after exporting GIT_DIR, rev-parse --show-toplevel seems to always just return the current directory.

The only thing, that I don’t understand is how/why did it work differently in the past.

So, I fixed this issue for me by playing around with the working directory of the git-lfs command. I have packaged my currently working solution into a Sphinx extension: https://github.com/ssciwr/sphinx_lfs_content Maybe having the code in a Sphinx extension will make it easier to report and address issues with the non-official LFS workflow on RTD.

I have the exact same problem and I can confirm what @rmol says. My log indicates that the LFS checkout was successful, still the files are pointers when the actual sphinx-build call happens. One thing I noticed is that since my last passing build and today, two additional stages were added to the build log (they show up as git rev-parse HEAD which would only be logging the HEAD SHA1, but may be more is happening at these stage). I can tell that the RTD-generated part of conf.py has not changed between the two build.