invoke: Using python3 keyword-only arguments or annotated arguments causes a ValueError from inspect

Since this issue is ignored i rolled out little monkeypatch. What it does is

• use getfullargspec which won’t raise exception
• translate it to same format that getargspec returns
• replace inspect.getargspec with patched argspec for duration of Task.getargspec call
• call original Task.getargspec

It needs only stdlib + invoke to work

# monkeypatch.py
from unittest.mock import patch
from inspect import getfullargspec, ArgSpec

import invoke


def fix_annotations():
    """
        Pyinvoke doesnt accept annotations by default, this fix that
        Based on: https://github.com/pyinvoke/invoke/pull/606
    """
    def patched_inspect_getargspec(func):
        spec = getfullargspec(func)
        return ArgSpec(*spec[0:4])

    org_task_argspec = invoke.tasks.Task.argspec

    def patched_task_argspec(*args, **kwargs):
        with patch(target="inspect.getargspec", new=patched_inspect_getargspec):
            return org_task_argspec(*args, **kwargs)

    invoke.tasks.Task.argspec = patched_task_argspec

at the top of your tasks.py do

from . import monkeypatch

monkeypatch.fix_annotations()

2.0.0 will be out soon & is now using inspect.signature!

I’ll revisit this after dropping Python 2, which is one of the next things on the roadmap. Should make it much easier to apply/fix overall.

If I declare the variable ‘c’ as Context, I wish set the type. But, this problem prevent me to type-hinting… I can’t understand why this is not fixed.

Any updates on this?

Heya, it looks like this still isn’t fixed? Any idea what the blockers are? It’s been years

For those playing along at home I am now working on issue and PR triage to help catch up on the backlog that has accumulated.

https://bitprophet.org/projects/#roadmap The big tasks that are being worked on are:

• Migrating CI to Circle CI
• Dropping Python2 support (that test matrix was getting wild)

Also invoke is part of the fabric and paramiko ecosystem and they need equal amounts of love too.

But I am doing my best over the coming weeks and months to help close the items that will have the greatest impact for our limited cognitive bandwidth and volunteer time.

Hi! I’m still seeing this with version 1.1.1.

nford@Nathaniels-MacBook-Pro:~/repos/deploy-apps $ invoke --list -v
Traceback (most recent call last):
  File "/usr/local/bin/invoke", line 11, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python3.6/site-packages/invoke/program.py", line 324, in run
    self.parse_collection()
  File "/usr/local/lib/python3.6/site-packages/invoke/program.py", line 408, in parse_collection
    self.load_collection()
  File "/usr/local/lib/python3.6/site-packages/invoke/program.py", line 599, in load_collection
    module, parent = loader.load(coll_name)
  File "/usr/local/lib/python3.6/site-packages/invoke/loader.py", line 76, in load
    module = imp.load_module(name, fd, path, desc)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 245, in load_module
    return load_package(name, filename)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/imp.py", line 217, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/nmford/repos/deploy-apps/tasks/__init__.py", line 2, in <module>
    from tasks import install
  File "/Users/nmford/repos/deploy-apps/tasks/install.py", line 26, in <module>
    def domino(context, app: str):
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 313, in task
    return klass(args[0], **kwargs)
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 77, in __init__
    self.positional = self.fill_implicit_positionals(positional)
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 168, in fill_implicit_positionals
    args, spec_dict = self.argspec(self.body)
  File "/usr/local/lib/python3.6/site-packages/invoke/tasks.py", line 154, in argspec
    spec = inspect.getargspec(func)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 1075, in getargspec
    raise ValueError("Function has keyword-only parameters or annotations"
ValueError: Function has keyword-only parameters or annotations, use getfullargspec() API which can support them

I am not sure what might need to be done to diagnose or resolve it from my end. Is anyone else still running into this?

@bitprophet Hello? How’s it going?

@karan-webkul Python 2 is at EOL, and while everyone using this package already is avoiding using type hints because it does not support it, this issue is around the fact that we’d like to both use the package and type hints, like other modern Python 3 apps.

Hi, still facing this issue, is there any update?

I do something similar to what @dralshehri showed, but for the function.

from invoke import Context, task


@task
def run(c):
    # type: (Context) -> None
    c.run("python src/main.py")

passing now.

Python 2 doesn’t support type hinting, So removing all the type hinting code will fix the issue.

Any action on this? Seems like using the inspect is now the way to go. Plz advise if this is something that can get resolved.

@nathanielford Well, the bug is more than 2 years old at the moment. Developers still didn’t apply a patch that is proposed by the exception message itself. So, they don’t care. You’re on your own.

@bitprophet - I would also like to contribute. I have previously worked on vcrpy as a maintainer for a year and my new role is seeing me need invoke a lot and I would like to put the time and energy into breathing new life into keeping invoke relevant. I couldn’t see a CONTRIBUTING.md to get a gist of who to contact about becoming a maintainer.

In particular I am coming from a few years over in Typescript where projects like lerna and npm workspaces coupled with the fact npm has scripts built into their package.json I feel this is lacking in the Python ecosystem.

poetry is about to launch v1.2.+ which allows plugins and I think combining poetry and invoke is a really awesome combination especially for multi-project mono-repos.

Especially as a consultant I can provide packages of prebaked tasks similar to the invocations project for our clients to get their dev workflows started.

If there is a gitter channel hit me up or DM me on LinkedIn or twitter.

We used invoke a lot, but now moving to https://typer.tiangolo.com/. I can really recommend it, it is kind of powerful and simple to use, comes with a lot of example documentation and elevates type hints!

Looks like this project is not maintained anymore, or does not care about Python 3 features 😕

@nathanielford Just run into this too. As it says in the exception itself, replaced inspect.getargspec at invoke/tasks.py with inspect.getfullargspec manually. Works for me.

@bitprophet - have you folks started working on this support? I’d be interested in contributing this support if not!

As a workaround for code completion to work in PyCharm,

I have to add type comments:

from invoke import context, task


@task
def hello(ctx, name):
    c = ctx  # type: context.Context
    c.run(f"echo 'Hello {name}'")

or sometimes I specify the type in docstrings:

from invoke import context, task


@task
def hello(c, name):
   """
   My docstring here.

   :type c: context.Context
   """
   c.run(f"echo 'Hello {name}'")

This might also work:

F = TypeVar('F', bound=Callable)

def task(f: F) -> F:
    f.__annotations__ = {}
    return invoke.task(f)

Haven’t tested it against anything but my current environment, so I don’t know if this will break MyPy or other type checkers, etc.

Thanks, @0az! Worked for me. Only 1 change to make MyPy happy:

F = TypeVar("F", bound=Callable[..., Any])

@italomaia and I encountered this using Fabric 2 with Python 3 (3.7.3 and 3.7.4 on Linux and macOS). Type hints in a Fabric task triggers ValueError complaining about getargspec: https://github.com/fabric/fabric/issues/1997

Several seemingly-easy solutions presented themselves over the years (since 2016?!) in PRs mentioning this issue. To recap:

1. Simplest is to use inspect.getfullargspec for Python 3: https://github.com/pyinvoke/invoke/pull/373

   diff --git a/invoke/tasks.py b/invoke/tasks.py
   index ed838c31..2ed82a2e 100644
   --- a/invoke/tasks.py
   +++ b/invoke/tasks.py
   @@ -11,8 +11,10 @@ from .util import six
   
   if six.PY3:
       from itertools import zip_longest
   +    getargspec = inspect.getfullargspec
   else:
       from itertools import izip_longest as zip_longest
   +    getargspec = inspect.getargspec
   
   from .context import Context
   from .parser import Argument, translate_underscores
   @@ -151,7 +153,7 @@ class Task(object):
           # TODO: __call__ exhibits the 'self' arg; do we manually nix 1st result
           # in argspec, or is there a way to get the "really callable" spec?
           func = body if isinstance(body, types.FunctionType) else body.__call__
   -       spec = inspect.getargspec(func)
   +       spec = getargspec(func)
           arg_names = spec.args[:]
           matched_args = [reversed(x) for x in [spec.args, spec.defaults or []]]
           spec_dict = dict(zip_longest(*matched_args, fillvalue=NO_DEFAULT))
   

2. Use invoke’s vendored module called decorator, which already did the heavy lifting: https://github.com/pyinvoke/invoke/pull/373#issuecomment-437580030

   diff --git a/invoke/tasks.py b/invoke/tasks.py
   index ed838c31..eaf0b622 100644
   --- a/invoke/tasks.py
   +++ b/invoke/tasks.py
   @@ -4,7 +4,7 @@ generate new tasks.
   """
   
   from copy import deepcopy
   -import inspect
   +from .vendor import decorator
   import types
   
   from .util import six
   @@ -151,7 +151,7 @@ class Task(object):
           # TODO: __call__ exhibits the 'self' arg; do we manually nix 1st result
           # in argspec, or is there a way to get the "really callable" spec?
           func = body if isinstance(body, types.FunctionType) else body.__call__
   -       spec = inspect.getargspec(func)
   +       spec = decorator.getargspec(func)
           arg_names = spec.args[:]
           matched_args = [reversed(x) for x in [spec.args, spec.defaults or []]]
           spec_dict = dict(zip_longest(*matched_args, fillvalue=NO_DEFAULT))
   

3. Or, since clearly decorator has given this more thought, which was the original holdup for https://github.com/pyinvoke/invoke/pull/373, just copy that block into invoke/task.py.

another workaround here.

monkey patch #373

import inspect
import six
import types
from invoke.tasks import Task, NO_DEFAULT

if six.PY3:
    from itertools import zip_longest
else:
    from itertools import izip_longest as zip_longest


def patched_argspec(self, body):
    """
    Returns two-tuple:
    * First item is list of arg names, in order defined.
        * I.e. we *cannot* simply use a dict's ``keys()`` method here.
    * Second item is dict mapping arg names to default values or
      `.NO_DEFAULT` (an 'empty' value distinct from None, since None
      is a valid value on its own).
    """
    # Handle callable-but-not-function objects
    # TODO: __call__ exhibits the 'self' arg; do we manually nix 1st result
    # in argspec, or is there a way to get the "really callable" spec?
    func = body if isinstance(body, types.FunctionType) else body.__call__
    if six.PY3:
        sig = inspect.signature(func)
        arg_names = [k for k, v in sig.parameters.items()]
        spec_dict = {}
        for k, v in sig.parameters.items():
            value = v.default if not v.default == sig.empty else NO_DEFAULT
            spec_dict.update({k: value})
    else:
        spec = inspect.getargspec(func)
        arg_names = spec.args[:]
        matched_args = [
            reversed(x) for x in [spec.args, spec.defaults or []]]
        spec_dict = dict(zip_longest(*matched_args, fillvalue=NO_DEFAULT))
    # Pop context argument
    try:
        context_arg = arg_names.pop(0)
    except IndexError:
        # TODO: see TODO under __call__, this should be same type
        raise TypeError("Tasks must have an initial Context argument!")
    del spec_dict[context_arg]
    return arg_names, spec_dict


Task.argspec = types.MethodType(patched_argspec, Task)

save above code to hotfix.py and import it when you need.

import hotfix
from invoke import task,Context

@task
def hello(c: Context):
    c.run('echo Hello world!')

@eruvanos We experimented with Typer as well, but missed the ability to run simple commands like invoke docs. With Typer, the equivalent would presumably look something like python -m tasks docs, which is much more verbose. Did you find a solution for that?