setuptools_scm: LookupError: setuptools-scm was unable to detect version (different case)

This is similar (but not the same as) https://github.com/pypa/setuptools_scm/issues/278

I have a private github repo hosting a number of packages, each of which is in an immediate subdirectory of the repo root. Each project has a similar setup.py. In some cases projects have dependencies on other projects from the same repo (but for the purposes of this problem I’m just considering one that does not).

The directory structure (from the repo root looks like this):

root
  c12e-data
    setup.py
    ...

The setup.py has content:

import os
from setuptools import setup

setup(
    name            = 'c12e_data',
    packages        = ['c12e.data'],
    use_scm_version = {
        'root'       : '..',
        'relative_to': os.path.dirname(__file__)
    },

    install_requires = [
        'typing_extensions'
    ],

    tests_require = [
        'hypothesis',
        'pytest >= 3.8.1, < 4.0',
        'c12e_testing'
    ],

    extras_require = {
        's3': ['boto3 >= 1.0.0, < 2.0.0']
    },

    setup_requires = [
        'setuptools_scm',
        'pytest-runner'
    ]
)

A tagged version of the repo has existed for several months and that tag has not changed. The tag concerned is v1.0.4. In ‘real life’ installation is via a conda environment.yml which pins pip and setuptools:

name: c12e-test
channels:
  - conda-forge
  - pytorch
  - defaults
dependencies:
  - pip=19.0.1
  - python=3.6.8
  - setuptools=40.7.3
  - toolz=0.9.0
  - scikit-learn
  - pip:
    - git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data

However for the purposes of minimal reproduction of the issue I can reproduce with just a pip install in an environment already set up with those versions of pip and setuptools using:

pip install  git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data

This fails with:

pip install  git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data
Collecting git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data
  Cloning ssh://git@github.com/CognitiveScale/c12e-python.git (to revision v1.0.4) to /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-zr1k52z7
Enter passphrase for key '/Users/sdraper/.ssh/github':
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-zr1k52z7/c12e_data/setup.py", line 28, in <module>
        'pytest-runner'
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/site-packages/setuptools/dist.py", line 447, in __init__
        k: v for k, v in attrs.items()
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/site-packages/setuptools/dist.py", line 739, in finalize_options
        ep.load()(self, ep.name, value)
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-zr1k52z7/c12e_data/.eggs/setuptools_scm-3.3.3-py3.6.egg/setuptools_scm/integration.py", line 17, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-zr1k52z7/c12e_data/.eggs/setuptools_scm-3.3.3-py3.6.egg/setuptools_scm/__init__.py", line 150, in get_version
        parsed_version = _do_parse(config)
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-zr1k52z7/c12e_data/.eggs/setuptools_scm-3.3.3-py3.6.egg/setuptools_scm/__init__.py", line 113, in _do_parse
        "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
    LookupError: setuptools-scm was unable to detect version for '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T'.

    Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

    For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj

I have run this with verbose output from pip and debug output from setuptools, but it wasn’t hugely revealing (to me anyway). Specifically:

SETUPTOOLS_SCM_DEBUG=1 pip install -vvv git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data
Created temporary directory: /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-ephem-wheel-cache-p27ef1wu
Created temporary directory: /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-tracker-d0aoaudl
Created requirements tracker '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-tracker-d0aoaudl'
Created temporary directory: /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-install-d3l9rxw6
Collecting git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data
  Created temporary directory: /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma
  Cloning ssh://git@github.com/CognitiveScale/c12e-python.git (to revision v1.0.4) to /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma
  Running command git clone -q ssh://git@github.com/CognitiveScale/c12e-python.git /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma
Enter passphrase for key '/Users/sdraper/.ssh/github':
  Running command git show-ref v1.0.4
  86662033d860a213121fe4deee2d85f0d01a7f16 refs/tags/v1.0.4
  Running command git rev-parse HEAD
  b4f9b49838d97bc51ad86617f19ce53074d08907
  Running command git checkout -q 86662033d860a213121fe4deee2d85f0d01a7f16
  Added git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data to build tracker '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-tracker-d0aoaudl'
    Running setup.py (path:/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data/setup.py) egg_info for package from git+ssh://git@github.com/CognitiveScale/c12e-python.git@v1.0.4#subdirectory=c12e_data
    Running command python setup.py egg_info
    root '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data'
    root '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma'
    root '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T'
    looking for ep setuptools_scm.parse_scm /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T
    looking for ep setuptools_scm.parse_scm_fallback /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data
    found ep pip-egg-info = setuptools_scm.hacks:parse_pip_egg_info
    pip-egg-info /private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data/pip-egg-info []
    found ep setup.py = setuptools_scm.hacks:fallback_version
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data/setup.py", line 28, in <module>
        'pytest-runner'
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/site-packages/setuptools/dist.py", line 447, in __init__
        k: v for k, v in attrs.items()
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/Users/sdraper/miniconda3/envs/c12e-python-recommender-examples/lib/python3.6/site-packages/setuptools/dist.py", line 739, in finalize_options
        ep.load()(self, ep.name, value)
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data/.eggs/setuptools_scm-3.3.3-py3.6.egg/setuptools_scm/integration.py", line 17, in version_keyword
        dist.metadata.version = get_version(**value)
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data/.eggs/setuptools_scm-3.3.3-py3.6.egg/setuptools_scm/__init__.py", line 150, in get_version
        parsed_version = _do_parse(config)
      File "/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T/pip-req-build-iq2g3xma/c12e_data/.eggs/setuptools_scm-3.3.3-py3.6.egg/setuptools_scm/__init__.py", line 113, in _do_parse
        "use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
    LookupError: setuptools-scm was unable to detect version for '/private/var/folders/pd/0s5cjm_13ddd1h72c75dskrwsm34nk/T'.

    Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

    For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
Cleaning up...

I have also tried running the steps pip takes manually (create temp dir, clone, checkout tag, cd to subdirectory, then run python setup.py egg_info and this worked (and created the egg just fine)!

Also I am 99.99% sure that exactly the same environment.yml was working a few weeks ago (and the tag definitely has not changed). Could there have been some transitive dependency change (in setuptools or pip) causing a regression or something? I cannot think what else might have changed.