tox: Tox 3.1 regression, ConfigError: substitution key not found

Thanks for submitting an issue!

If submitting a BUG please provide:

• Minimal reproducible example or detailed description, assign “bug”
• OS and pip list output

Reproducible, testing on Linux CentOS with Python via conda:

$ which python
/mnt/shared/scratch/xxx/apps/conda/bin/python
$ python --version
Python 2.7.15
$ python -m pip list
Package                       Version    
----------------------------- -----------
absl-py                       0.1.10     
argh                          0.26.1     
asn1crypto                    0.24.0     
astor                         0.6.2      
avro                          1.8.0      
backports.functools-lru-cache 1.5        
backports.weakref             1.0rc1     
biopython                     1.68       
bleach                        1.5.0      
certifi                       2018.4.16  
cffi                          1.11.5     
chardet                       3.0.4      
codecov                       2.0.15     
conda                         4.5.5      
ConsensusCore                 1.0.2      
coverage                      4.5.1      
cryptography                  2.2.1      
cycler                        0.10.0     
Cython                        0.28.2     
decorator                     4.3.0      
enum34                        1.1.6      
funcsigs                      1.0.2      
functools32                   3.2.3.post2
future                        0.16.0     
futures                       3.2.0      
gast                          0.2.0      
GenomicConsensus              1.1.0      
grpcio                        1.11.0     
h5py                          2.7.0      
html5lib                      0.9999999  
idna                          2.6        
ipaddress                     1.0.22     
iso8601                       0.1.12     
jsonschema                    2.6.0      
Keras                         2.1.5      
kiwisolver                    1.0.1      
Mako                          1.0.7      
Markdown                      2.6.11     
MarkupSafe                    1.0        
matplotlib                    2.2.2      
mkl-fft                       1.0.2      
mkl-random                    1.0.1      
mmtf-python                   1.0.2      
mock                          2.0.0      
msgpack                       0.5.6      
msgpack-python                0.5.6      
ncbi-genome-download          0.2.6      
networkx                      1.11       
numpy                         1.14.3     
olefile                       0.45.1     
packaging                     17.1       
pandas                        0.22.0     
pathtools                     0.1.2      
pbcommand                     0.3.29     
pbcore                        1.2.10     
pbr                           4.0.2      
Pillow                        5.0.0      
pip                           10.0.1     
pluggy                        0.6.0      
protobuf                      3.5.2      
py                            1.5.4      
pycosat                       0.6.3      
pycparser                     2.18       
pygpu                         0.7.5      
pyOpenSSL                     17.5.0     
pyparsing                     2.2.0      
pypeflow                      2.0.2      
pysam                         0.14.1     
PySocks                       1.6.8      
python-dateutil               2.3        
pytz                          2017.3     
PyYAML                        3.12       
reportlab                     3.4.0      
requests                      2.13.0     
ruamel-yaml                   0.15.35    
scipy                         1.1.0      
setuptools                    39.1.0     
six                           1.11.0     
subprocess32                  3.5.1      
tensorboard                   1.6.0      
tensorflow                    1.6.0      
termcolor                     1.1.0      
Theano                        1.0.1      
tox                           3.0.0      
urllib3                       1.12       
virtualenv                    16.0.0     
watchdog                      0.8.3      
webencodings                  0.5        
Werkzeug                      0.14.1     
wheel                         0.31.0     
xmlbuilder                    1.0 

Creating test case:

$ cd /tmp
$ mkdir tox-regression
$ cd tox-regression
$ curl -L -o tox.ini https://github.com/biopython/biopython/raw/biopython-172/.travis-tox.ini

Using tox v3.0.0 works,

$ python -m pip install tox==3.0.0
...
$ python -m tox --version
3.0.0 imported from /mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/__init__.pyc
$ rm -rf .tox
$ python -m tox -e py27-nocov --notest
py27-nocov create: /tmp/tox-regression/.tox/py27-nocov
py27-nocov installdeps: unittest2, mysql-python, mmtf-python, reportlab, psycopg2-binary, mysql-connector-python-rf, rdflib, numpy, networkx
py27-nocov installed: aenum==1.4.5,args==0.1.0,avro==1.8.1,awscli==1.11.41,backports.lzma==0.0.1,beautifulsoup4==4.5.1,bioblend==0.10.0,biopython==1.71,boto==2.38.0,botocore==1.5.4,CacheControl==0.11.7,click==4.0,clint==0.5.1,colorama==0.3.7,configparser==3.5.0,cwltool==1.0.20170828135420,decorator==4.3.0,docutils==0.12,enum34==1.1.6,ephemeris==0.4.0,epydoc==3.0.1,falcon-kit==1.0+git.f20989451b7e6826bcedf8729ad4b56237bc8b83,flake8==3.2.1,flake8-import-order==0.11,funcsigs==0.4,futures==3.0.5,galaxy-lib==17.9.10,GenomeTools==0.1,gff3toembl==1.1.2,glob2==0.4.1,gxformat2==0.1.1,html5lib==1.0b3,isodate==0.5.4,Jinja2==2.7.3,jmespath==0.9.0,keepalive==0.5,khmer==0.4,linecache2==1.0.0,lockfile==0.12.2,lxml==3.6.4,MarkupSafe==0.23,matplotlib==1.1.1,mccabe==0.5.2,mistune==0.7.3,mmtf-python==1.0.5,mock==1.3.0,msgpack-python==0.4.8,mysql-connector-python==2.0.4,mysql-connector-python-rf==2.2.2,MySQL-python==1.2.5,networkx==2.1,nose==1.2.1,numpy==1.14.2,olefile==0.44,pandas==0.22.0,pbr==1.8.1,pep8==1.7.0,Pillow==4.1.0,pkginfo==1.3.2,planemo==0.48.0.dev0,pluggy==0.3.1,poster==0.8.1,psycopg2-binary==2.7.5,py==1.4.30,pyaml==16.11.4,pyani==0.1.3.2,pyasn1==0.1.9,pycodestyle==2.2.0,pydablooms==0.8.2,pyflakes==1.3.0,PyGithub==1.25.2,pyparsing==2.1.5,pytest==3.0.5,python-dateutil==2.5.1,pytz==2016.3,PyYAML==3.11,rdflib==4.2.2,rdflib-jsonld==0.4.0,reportlab==3.4.0,requests==2.10.0,requests-toolbelt==0.7.0,rpy2==2.7.0,rsa==3.4.2,ruamel.ordereddict==0.4.9,ruamel.yaml==0.14.8,s3transfer==0.1.10,schema-salad==2.6.20171201034858,scipy==0.17.1,screed==0.7,seaborn==0.7.0,shellescape==3.4.1,singledispatch==3.4.0.3,six==1.10.0,SPARQLWrapper==1.7.6,tox==2.1.1,traceback2==1.4.0,twine==1.8.1,typing==3.6.4,unittest2==1.1.0,virtualenv==13.1.2
__________________________________________________________________ summary __________________________________________________________________
  py27-nocov: skipped tests
  congratulations :)

Now trying with the latest tox,

$ python -m pip install tox==3.1.1
...
$ python -m tox --version
3.1.1 imported from /mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/__init__.pyc
$ rm -rf .tox
$ python -m tox -e py27-nocov --notest
Traceback (most recent call last):
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/__main__.py", line 4, in <module>
    tox.cmdline()
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/session.py", line 39, in cmdline
    main(args)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/session.py", line 44, in main
    config = prepare(args)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/session.py", line 26, in prepare
    config = parseconfig(args)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 246, in parseconfig
    parseini(config, inipath)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 975, in __init__
    config.envconfigs[name] = self.make_envconfig(name, section, reader._subs, config)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1012, in make_envconfig
    res = reader.getlist(env_attr.name, sep="\n")
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1156, in getlist
    s = self.getstring(name, None)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1218, in getstring
    x = self._replace_if_needed(x, name, replace, crossonly)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1261, in _replace_if_needed
    x = self._replace(x, name=name, crossonly=crossonly)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1248, in _replace
    replaced = Replacer(self, crossonly=crossonly).do_replace(value)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1289, in do_replace
    expanded = substitute_once(value)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1287, in substitute_once
    return self.RE_ITEM_REF.sub(self._replace_match, x)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1330, in _replace_match
    return self._replace_substitution(match)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1365, in _replace_substitution
    val = self._substitute_from_other_section(sub_key)
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1359, in _substitute_from_other_section
    raise tox.exception.ConfigError("substitution key {!r} not found".format(key))
tox.exception.ConfigError: ConfigError: substitution key 'py27' not found

Interestingly trying to use the other Python versions, e.g. py36, also gives this warning about py27,

$ python -m tox -e py36-nocov --notest
Traceback (most recent call last):
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
...
  File "/mnt/shared/scratch/xxx/apps/conda/lib/python2.7/site-packages/tox/config.py", line 1359, in _substitute_from_other_section
    raise tox.exception.ConfigError("substitution key {!r} not found".format(key))
tox.exception.ConfigError: ConfigError: substitution key 'py27' not found

Note there is no explicit usage of baseversion in this configuration file, which starts:

[tox]
minversion = 2.0
skipsdist = True
envlist =
    style
    sdist
    bdist_wheel
    {py27,py34,py35,py36,pypy,pypy3}-cover
    {py27,py34,py35,py36,pypy,pypy3}-nocov

[testenv]
# TODO: Try tox default sdist based install instead:
skip_install = True
sitepackages = True
passenv =
    CI
    TRAVIS
    TRAVIS_*
    TOXENV
    CODECOV_*
whitelist_externals =
    bash
    echo
# Want to avoid overhead of compiling numpy or scipy:
# (But must compile numpy for PyPy right now)
install_command = pip install --only-binary=scipy {opts} {packages}
deps =
    #Lines startings xxx: are filtered by the environment.
    #Leaving py34 without any soft dependencies (just numpy)
    cover: coverage
    cover: codecov
    {py27}: unittest2
    {py27}: mysql-python
    {py27,py36}: mmtf-python
    {py27,py35}: reportlab
    {py27,py34,py35,py36}: psycopg2-binary
    {py27,py34,py35,py35}: mysql-connector-python-rf
    {py27,py35,pypy}: rdflib
    {pypy,pypy3}: numpy==1.12.1
    {py27,py34,py36}: numpy
    {py36}: scipy
    {py27}: networkx
    {py36}: matplotlib
commands =
    #The bash call is a work around for special characters
    #The /dev/null is to hide the verbose output but leave warnings
    bash -c \'python setup.py install > /dev/null\'
    #The bash call is a work around for the cd command
    nocov: bash -c \'cd Tests && python run_tests.py --offline\'
    #See https://codecov.io/ and https://github.com/codecov/example-python
    cover: bash -c \'rm -rf Tests/coverage.xml\'
    cover: bash -c \'cd Tests && coverage run run_tests.py --offline && coverage xml\'
    cover: codecov --file Tests/coverage.xml -X pycov -X gcov

...

I suspect it is linked to how we have environments py27-nocov and py27-cover setup (and likewise for the other versions) to define testing with and without coverage tracking, both using py27 for their dependencies.