pytest: "unrecognized arguments" or "option names already added" error when running pytest inside a Docker container using Jenkins

This is a complicated set up, so bear with me.

I am trying to create a Jenkins pipeline job to run Pytest on Jenkins inside a Docker container. As far as I can tell, when Jenkins starts a Docker container, it does so with the working directory of that container set to a directory on the Jenkins filesystem, under /var/jenkins_home/<job name>/ by default. Thus, running the tests requires:

1. Pull the Docker image, if necessary.
2. Create the container using the image. (This gives me access to all the test dependencies, which are already installed inside the image.)
3. Checkout the current source code for the tests from GIthub.
4. Run pytest.
5. Have Jenkins process the resulting jUnit XML.

Because Jenkins passes job parameters as environment variables and some other probably-not-relevant reasons, I need to use a front-end script, run_tests.py, to turn those variables into command-line arguments that pytest can understand. run_tests.py then calls pytest.main().

    pytest.main(pytest_args + test_args)

Here, pytest_args are arguments for pytest itself like -s, -k, or --tb=<type>. test_args are command-line arguments that I have configured in conftest.py using pytest_addoption() : these are the arguments created from environment variables. The layout of the project looks like:

repo_root/
repo_root/test_automation/
repo_root/test_automation/conftest.py
repo_root/test_automation/run_tests.py
repo_root/test_automation/test_files.py

This setup works outside Jenkins: I cd to test_automation/, run run_tests.py, it finds conftest.py, and everything is happy. When I run the Jenkins job that’s supposedly executing this same steps with PYTEST_DEBUG set:

12:30:51     pytest_load_initial_conftests [hook]
12:30:51         early_config: <_pytest.config.Config object at 0x7f8e2f160c88>
12:30:51         args: [...]
12:30:51         parser: <_pytest.config.Parser object at 0x7f8e2f1717f0>
12:30:51       pytest_plugin_registered [hook]
12:30:51           plugin: <_pytest.capture.CaptureManager object at 0x7f8e2d6b2438>
12:30:51           manager: <_pytest.config.PytestPluginManager object at 0x7f8e2f5a5d30>
12:30:51       finish pytest_plugin_registered --> [] [hook]
12:30:51     finish pytest_load_initial_conftests --> [] [hook]
12:30:51     pytest_cmdline_preparse [hook]
12:30:51         config: <_pytest.config.Config object at 0x7f8e2f160c88>
12:30:51         args: [...]
12:30:51     finish pytest_cmdline_preparse --> [] [hook]
12:30:51 usage: run_tests.py [options] [file_or_dir] [file_or_dir] [...]
12:30:51 run_tests.py: error: unrecognized arguments: ...

Here, the “…” for the args stands for all the arguments in the first two cases, including the pytest-specific ones, and only the arguments configured in conftest.py for the unrecognized arguments, the last case. From looking at the logs for this same configuration that works outside Jenkins, I know that the pytest_load_initial_conftests log should look like:

    pytest_load_initial_conftests [hook]
        early_config: <_pytest.config.Config object at 0x10f512208>
        args: [...]
        parser: <_pytest.config.Parser object at 0x10f512d30>
      pytest_plugin_registered [hook]
          plugin: <_pytest.capture.CaptureManager object at 0x1101ec5f8>
          manager: <_pytest.config.PytestPluginManager object at 0x10f2caba8>
      finish pytest_plugin_registered --> [] [hook]
    find_module called for: conftest [assertion]
    rewriting conftest file: '/Users/user/test_automation/test_automation/conftest.py' [assertion]
    found cached rewritten pyc for '/Users/user/test_automation/test_automation/conftest.py' [assertion]
    find_module called for: run_tests [assertion]
    loaded conftestmodule <module 'conftest' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x11019c198>)> [pluginmanage]
      pytest_addoption [hook]
          parser: <_pytest.config.Parser object at 0x10f512d30>
          __multicall__: <_MultiCall 0 results, 0 meths, kwargs={'parser': <_pytest.config.Parser object at 0x10f512d30>, '__multicall__': <_MultiCall 0 results, 0 meths, kwargs={...}>}>
      finish pytest_addoption --> [] [hook]
      pytest_plugin_registered [hook]
          plugin: <module 'conftest' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x11019c198>)>
          manager: <_pytest.config.PytestPluginManager object at 0x10f2caba8>
      finish pytest_plugin_registered --> [] [hook]
    finish pytest_load_initial_conftests --> [] [hook] 

It looks a lot like pytest is not finding my conftest.py file, even though it’s in the root directory (with respect to how pytest is being run). However, if I have Jenkins run python3 -m pytest --help in the same directory, with everything else the same, pytest will correctly process conftest.py and display the options that are configured there under the custom options: heading. I also tried manually forcing pytest to load conftest.py using -p. When I do that, pytest crashes with a traceback that suggests that conftest.py is already loaded:

14:24:15     pytest_load_initial_conftests [hook]
14:24:15         early_config: <_pytest.config.Config object at 0x7f5acc971780>
14:24:15         args: [...]
14:24:15         parser: <_pytest.config.Parser object at 0x7f5acc9802e8>
14:24:15       pytest_plugin_registered [hook]
14:24:15           plugin: <_pytest.capture.CaptureManager object at 0x7f5acab0ac50>
14:24:15           manager: <_pytest.config.PytestPluginManager object at 0x7f5acc5ff4e0>
14:24:15       finish pytest_plugin_registered --> [] [hook]
14:24:15     find_module called for: conftest [assertion]
14:24:15     rewriting conftest file: '/var/jenkins_home/workspace/Test Automation-pipeline/test_automation/conftest.py' [assertion]
14:24:15     found cached rewritten pyc for '/var/jenkins_home/workspace/Test Automation-pipeline/test_automation/conftest.py' [assertion]
14:24:15     loaded conftestmodule <module 'conftest' (<_pytest.assertion.rewrite.AssertionRewritingHook object at 0x7f5acabb7668>)> [pluginmanage]
14:24:15       pytest_addoption [hook]
14:24:15           parser: <_pytest.config.Parser object at 0x7f5acc9802e8>
14:24:15           __multicall__: <_MultiCall 0 results, 0 meths, kwargs={'parser': <_pytest.config.Parser object at 0x7f5acc9802e8>, '__multicall__': <_MultiCall 0 results, 0 meths, kwargs={...}>}>
14:24:15 Traceback (most recent call last):
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 342, in _getconftestmodules
14:24:15     return self._path2confmods[path]
14:24:15 KeyError: local('/var/jenkins_home/workspace/Test Automation-pipeline/test_automation')
14:24:15 
14:24:15 During handling of the above exception, another exception occurred:
14:24:15 
14:24:15 Traceback (most recent call last):
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 373, in _importconftest
14:24:15     return self._conftestpath2mod[conftestpath]
14:24:15 KeyError: local('/var/jenkins_home/workspace/Test Automation-pipeline/test_automation/conftest.py')
14:24:15 
14:24:15 During handling of the above exception, another exception occurred:
14:24:15 
14:24:15 Traceback (most recent call last):
14:24:15   File "run_tests.py", line 435, in <module>
14:24:15     invoke_pytest(args, pytest_args)
14:24:15   File "run_tests.py", line 201, in invoke_pytest
14:24:15     pytest.main(pytest_args + test_args)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 49, in main
14:24:15     config = _prepareconfig(args, plugins)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 168, in _prepareconfig
14:24:15     pluginmanager=pluginmanager, args=args)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
14:24:15     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
14:24:15     return self._inner_hookexec(hook, methods, kwargs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 302, in __call__
14:24:15     return outcome.get_result()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
14:24:15     raise ex[1].with_traceback(ex[2])
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
14:24:15     self.result = func()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 300, in <lambda>
14:24:15     outcome = _CallOutcome(lambda: self.oldcall(hook, hook_impls, kwargs))
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
14:24:15     _MultiCall(methods, kwargs, hook.spec_opts).execute()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
14:24:15     return _wrapped_call(hook_impl.function(*args), self.execute)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
14:24:15     wrap_controller.send(call_outcome)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/helpconfig.py", line 68, in pytest_cmdline_parse
14:24:15     config = outcome.get_result()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
14:24:15     raise ex[1].with_traceback(ex[2])
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
14:24:15     self.result = func()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
14:24:15     res = hook_impl.function(*args)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 945, in pytest_cmdline_parse
14:24:15     self.parse(args)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 1116, in parse
14:24:15     self._preparse(args, addopts=addopts)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 1087, in _preparse
14:24:15     args=args, parser=self._parser)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
14:24:15     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
14:24:15     return self._inner_hookexec(hook, methods, kwargs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 302, in __call__
14:24:15     return outcome.get_result()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
14:24:15     raise ex[1].with_traceback(ex[2])
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
14:24:15     self.result = func()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 300, in <lambda>
14:24:15     outcome = _CallOutcome(lambda: self.oldcall(hook, hook_impls, kwargs))
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
14:24:15     _MultiCall(methods, kwargs, hook.spec_opts).execute()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
14:24:15     return _wrapped_call(hook_impl.function(*args), self.execute)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
14:24:15     return call_outcome.get_result()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
14:24:15     raise ex[1].with_traceback(ex[2])
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
14:24:15     self.result = func()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
14:24:15     res = hook_impl.function(*args)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 991, in pytest_load_initial_conftests
14:24:15     self.pluginmanager._set_initial_conftests(early_config.known_args_namespace)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 328, in _set_initial_conftests
14:24:15     self._try_load_conftest(current)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 331, in _try_load_conftest
14:24:15     self._getconftestmodules(anchor)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 356, in _getconftestmodules
14:24:15     mod = self._importconftest(conftestpath)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 392, in _importconftest
14:24:15     self.consider_conftest(mod)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 415, in consider_conftest
14:24:15     self.register(conftestmodule, name=conftestmodule.__file__)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 264, in register
14:24:15     ret = super(PytestPluginManager, self).register(plugin, name)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 371, in register
14:24:15     hook._maybe_apply_history(hookimpl)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 768, in _maybe_apply_history
14:24:15     res = self._hookexec(self, [method], kwargs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
14:24:15     return self._inner_hookexec(hook, methods, kwargs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 302, in __call__
14:24:15     return outcome.get_result()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
14:24:15     raise ex[1].with_traceback(ex[2])
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
14:24:15     self.result = func()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 300, in <lambda>
14:24:15     outcome = _CallOutcome(lambda: self.oldcall(hook, hook_impls, kwargs))
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
14:24:15     _MultiCall(methods, kwargs, hook.spec_opts).execute()
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
14:24:15     res = hook_impl.function(*args)
14:24:15   File "/var/jenkins_home/workspace/Test Automation-pipeline/test_automation/conftest.py", line 19, in pytest_addoption
14:24:15     parser.addoption(option, *run_tests.TEST_OPTIONS[option][0], **run_tests.TEST_OPTIONS[option][1])
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 537, in addoption
14:24:15     self._anonymous.addoption(*opts, **attrs)
14:24:15   File "/usr/local/lib/python3.6/site-packages/_pytest/config.py", line 762, in addoption
14:24:15     raise ValueError("option names %s already added" % conflict)
14:24:15 ValueError: option names {'--<option'} already added

I tried passing the --debug option to pytest, but it also crashed with this traceback, so it didn’t give me any more information.

• pip list from inside the container:

boltons (17.2.0)
cachetools (2.0.1)
certifi (2017.11.5)
chardet (3.0.4)
ConfigArgParse (0.12.0)
decorator (4.1.2)
dictdiffer (0.7.0)
docker (2.7.0)
docker-pycreds (0.2.1)
dpath (1.4.0)
google-auth (1.2.1)
idna (2.6)
ipaddress (1.0.19)
Jinja2 (2.10)
jsonschema (2.6.0)
kubernetes (3.0.0)
MarkupSafe (1.0)
networkx (2.0)
openshift (0.3.4)
pip (9.0.1)
plumbum (1.6.5)
py (1.5.2)
pyasn1 (0.4.2)
pyasn1-modules (0.2.1)
pytest (3.2.0)
pytest-html (1.16.1)
pytest-metadata (1.5.1)
python-dateutil (2.6.1)
python-string-utils (0.6.0)
PyYAML (3.12)
requests (2.18.4)
rsa (3.4.2)
ruamel.yaml (0.15.35)
setuptools (36.6.0)
six (1.11.0)
swagger-parser (1.0.0)
swagger-spec-validator (2.1.0)
urllib3 (1.22)
websocket-client (0.40.0)
wheel (0.30.0)

• I’m using Docker for Mac to run Docker on Mac OS X 10.12.6. (Docker for Mac uses Virtualbox to create a VM on Mac OS X.) Docker is version Version 17.09.1-ce-mac42 (21090).

• Jenkins is also running in a Docker container, and from what I see the Docker socket is mounted inside Jenkins, so there’s only one Docker instance running. Jenkins is version 2.89.2.

Is there anything else I can do to figure out what’s going on here? I haven’t been able to replicate the bug anywhere outside a Jenkins job. Even when I have Jenkins start the container, then use docker exec to log into, I can still run run_tests.py and it doesn’t complain about unrecognized arguments.