vscode-python: Debugger doesn't stop at breakpoints with pytest if pytest-cov is used

@lmazuel apologies for the delay. Looks like its not possible to debug the code with coverage enabled (http://pytest-cov.readthedocs.io/en/latest/debuggers.html). I guess we’ll have to disable ‘coverage’ on the fly when debugging (using the flag "--no-cov").

I use an alternative to @dferrante 's solution above, which seems to be seamless inside the VSCode IDE for pytest tests:

In setup.cfg I have:

[tool:pytest]
addopts =  --cov=<module name> --cov-report term --cov-report xml:cov.xml  <other arguments here>

And then in launch.json I define a new task:

 {
    "name": "Debug Unit Test",
    "type": "python",
    "request": "launch",
    "justMyCode": false,
    "program": "${file}",
    "purpose": ["debug-test"],
    "console": "integratedTerminal",
    "env": {
        "PYTEST_ADDOPTS": "--no-cov"
    },
}

When you press the “Debug Test” button in the testing panel or by right-clicking on a test itself, VSCode executes this task which just turns off all coverage for that run. No need to switch between configurations or altering config arguments on the fly.

Note this may only work for fairly new versions of VSCode - I’m on 1.60.

@DonJayamanne I don’t really like having to edit my workspace settings any time I want to switch back and forth between generating coverage and debugging. What about the idea of specifying an extra set of arguments that would be appended to python.unitTest.pyTestArgs when the “debug unit test” feature is used? It’d give users a place to put “–no-cov”, if they happen to be using coverage. Maybe call it python.unitTest.pyTestDebugExtraArgs?

I know this is old, but know that since Python 3.7 you can use a built-in breakpoint, and it works fine with cov.

just insert this in your code

breakpoint()

There’s no straightforward (and maintainable) way that the coverage tracker can know if the parts of your code that the debug terminal invoked should count towards coverage or not. The coverage frameworks disable breakpoints so that they can get accurate coverage reporting. It’s for this reason that I suggested a new settings.json namespace above so that we can specify different arguments for test debugging (i.e. with coverage disabled)

Wouldn’t the cleanest solution be for VSCode to automatically disable code coverage when a test is run in debug mode? User side you would see no code coverage when you run tests in debug but code coverage appear fine if you run without debug. That would be pretty clean and painless for the two scenarios (rather than having to fuss with run settings and env variables).

@lmazuel Unfortunately we’re not going to be able to resolve this easily, hence we’re closing this. My suggestion is to modify the pytest args in the settings.json to manually pass in the --no-cov flag to enable debugging. I understand this isn’t the best solution, as you’ll be modifying the settings on and off.

@Limonkufu Put simply, there’s no fix. Think about it this way:

1. Coverage is enabled and tests are run in debug mode
2. A breakpoint is hit and the program is paused
3. You run a command in the debug terminal that calls parts of your code

There’s no straightforward (and maintainable) way that the coverage tracker can know if the parts of your code that the debug terminal invoked should count towards coverage or not. The coverage frameworks disable breakpoints so that they can get accurate coverage reporting. It’s for this reason that I suggested a new settings.json namespace above so that we can specify different arguments for test debugging (i.e. with coverage disabled)

@skilkis I tested this with launch.json vs. the “launch” setting in settings.json.

# settings.json
// REDACTED
    "launch": {
        "configurations": [
            {
                "name": "Python: Global Debug Tests",
                "type": "python",
                "request": "launch",
                "program": "${file}",
                "purpose": ["debug-test"],
                "console": "integratedTerminal",
                "justMyCode": true,
                "env": {"PYTEST_ADDOPTS": "--no-cov"}
            }
        ],
        "compounds": []
    },
// REDACTED

vs.

# launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Debug Tests",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "purpose": ["debug-test"],
            "console": "integratedTerminal",
            "justMyCode": true,
            "env": {"PYTEST_ADDOPTS": "--no-cov"}
        }
    ]
}

Both options show up in the “Run and Debug” sidebar and work as expected if selected. But if I comment out the contents of launch.json or delete the file, the “Python: Global Debug Tests” option will still be there but will not apply the env variable meaning the debugger will not work correctly. This is definitely a bug with the “launch” option vs. launch.json behavior.

You can recreate this with any tests.

@skilkis I use multi-root workspaces. launch.json isn’t part of that workflow. The launch section of the workspace file should be treated the same as a launch.json (and if it’s not being treated the same, that’s a bug).

I can confirm that debugging in 1.68.0-insider doesn’t work with this workspace (see below). You can see that all of the relevant settings in the launch section of this multi-root workspace are the same as @skilkis’s settings.

{
    "folders": [
        {
            "path": "../../../repos/repo-a"
        },
        {
            "path": "../../../repos/repo-b"
        }
    ],
    "launch": {
        "compounds": [],
        "configurations": [
            {
                "console": "integratedTerminal",
                "env": {
                    "PYTEST_ADDOPTS": "--no-cov"
                },
                "justMyCode": false,
                "name": "Python: Debug",
                "purpose": [
                    "debug-test"
                ],
                "request": "launch",
                "type": "python"
            },
            {
                "console": "integratedTerminal",
                "env": {
                    "PYTEST_ADDOPTS": "--no-cov"
                },
                "name": "Python: Current File",
                "program": "${file}",
                "request": "launch",
                "type": "python"
            },
            {
                "console": "integratedTerminal",
                "cwd": "${fileWorkspaceFolder}",
                "env": {
                    "PYTEST_ADDOPTS": "--no-cov"
                },
                "name": "Python: Current File in Current Workspace",
                "program": "${file}",
                "request": "launch",
                "type": "python"
            }
        ],
        "version": "0.2.0"
    },
    "settings": {
        "debug.internalConsoleOptions": "openOnFirstSessionStart",
        "files.trimTrailingWhitespace": true,
        "python.defaultInterpreterPath": "~/virtualenvs/dev/bin/python3",
        "python.testing.autoTestDiscoverOnSaveEnabled": true,
        "python.testing.pytestEnabled": true,
        "python.testing.unittestArgs": [
            "-v",
            "-s",
            ".",
            "-p",
            "*test*.py",
            "-f"
        ],
        "python.testing.unittestEnabled": true,
        "terminal.integrated.profiles.linux": {
            "bash": {
                "args": [
                    "-l"
                ],
                "path": "/bin/bash"
            }
        }
    }
}

The only way to make the debugger work is to remove addopts = --cov repo-a ... from your pyproject.toml, setup.cfg, etc.

If it works in a launch.json as @skilkis claims, but not in a workspace file, that’s a bug.

for those coming to this thread, i have a solution that may work for some people. it involves moving your pytest args into pytest.ini within your project, setting up two tasks to sed and un-sed the coverage args, and adding these tasks to your launch.json.

sample pytest.ini (this is for a django project):

[pytest]
addopts = --cov-config=.coveragerc --cov-report html --cov-report xml --cov=. --nomigrations --reuse-db

tasks in tasks.json that use sed to overwrite the pytest.ini:

        {
            "label": "coverage-off",
            "type": "shell",
            "command": "sed -i 's/addopts = --cov-config=.coveragerc --cov-report html --cov-report xml --cov=./addopts =/' pytest.ini",
            "problemMatcher": []
        },
        {
            "label": "coverage-on",
            "type": "shell",
            "command": "sed -i 's/addopts =/addopts = --cov-config=.coveragerc --cov-report html --cov-report xml --cov=./' pytest.ini",
            "problemMatcher": []
        }

launch.json for debug tests:

        {
            "name": "Debug Tests",
            "type": "python",
            "preLaunchTask": "coverage-off",
            "justMyCode": false,
            "postDebugTask": "coverage-on",
            "request": "test",
            "console": "integratedTerminal",
        }

slightly annoying solution as it pops open a task window in the terminal, but hey, it works. switch in your own pytest args, or if its only coverage, you can use sed to just comment out the line.

This is actually worse than it seems at first blush. Not only will running with coverage enabled fail to respect breakpoints in PyCharm, but I’ve also found it causes manually inserted breakpoint() lines to stop at locations that are unpredictable and not the specified breakpoint. Not even close. This was with using the arguments of --cov and --cov-report=term.

@ericchansen I’ve created a minimal reproduction of your configuration and it also does not work for me. The code I used to test it is located at skilkis/vscode_python_pytest_multi_root_breakpoint_issue. Would you mind testing the launch.json file I linked above in a non multi-root workspace? This so that we can confirm that it works fine on your system.

After this I believe we can make a feature request to have support for settings.json and multi-root workspaces!

@sgbaird here is the launch.json that resolved the issue for me. I’ve hidden it from view to not clutter the debug configuration menu 😊

{
  "version": "0.2.0",
  "configurations": [
    {
      // Disable cov to allow breakpoints when launched from VS Code Python
      "name": "Python: Debug Tests",
      "type": "python",
      "request": "launch",
      "program": "${file}",
      "purpose": ["debug-test"],
      "console": "internalConsole",
      "justMyCode": false,
      "presentation": {
        "hidden": true
      },
      "env": {
        "PYTEST_ADDOPTS": "--no-cov"
      }
    }
  ]
}

EDIT: Original fix was provided by @AlexanderWells-diamond in the answer above

Its something in vscode. If you right-click your test in “testing” then select Debug from the menu it works fine and you can hit your breakpoints in “testing”. The debug “button” on the test does not allow the process to stop on a breakpoint.

Thanks for the --no-cov trick ! I have to edit my configuration anytime I want to switch between coverage and debug, so I hope someone develops the enhancement proposed by @justfalter

Since I don’t know how to develop VS Code plugins, I will use the trick for now 😃