serverless-python-requirements: Private package fails with `error: option --home not recognized`

I have a private repo on Github with a python package located under a subdirectory sdk/ that I can pip install locally with:

pip install -e git+ssh://git@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

Trying to add this to the requirments.txt, it fails with a permission denied - even with adding:

custom:
  pythonRequirements:
    dockerizePip: true
    dockerSsh: true

So I tried using a Git token to get around SSH issue for now; again it works locally with:

pip install -e git+ssh://<access-token>@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

However, now I am getting the error error: option --home not recognized

Output from SLS_DEBUG=* sls deploy --stage=dev:

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/requirements.txt in /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Obtaining sdk from git+https://<access-token>@github.com/<org>/<repo>.git@master#egg=sdk&subdirectory=sdk (from -r /var/task/requirements.txt (line 1))
  Cloning https://<access-token>@github.com/<org>/<repo>.git to ./src/sdk
Installing collected packages: sdk
  Running setup.py develop for sdk
    Complete output from command /var/lang/bin/python3.6 -c "import setuptools, tokenize;__file__='/var/task/src/sdk/sdk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-zkul1ujz:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --home not recognized

    ----------------------------------------
Command "/var/lang/bin/python3.6 -c "import setuptools, tokenize;__file__='/var/task/src/sdk/sdk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-zkul1ujz" failed with error code 1 in /var/task/src/sdk/sdk

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:247:11)
    at installRequirementsIfNeeded (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:423:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:502:29)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.11.4
     Serverless Version:     1.31.0

For reference, my setup.py:

#!/usr/bin/env python
"""
    Setup
    -----
    Enables the `setuptools` to build and install the library
"""

from setuptools import setup

packages = ['sdk']

requires = [
    'boto3~=1.9.2',
    'botocore~=1.12.2',
    'dnspython~=1.15.0',
    'email_validator~=1.0.3',
    'idna~=2.6',
    'requests~=2.19.1'
]
test_requirements = [
    'freezegun~=0.3.10',
    'pytest~=3.5.0',
    'pytest-cov'
]


with open('README.md', 'r', 'utf-8') as rfile:
    readme = rfile.read()

setup(
    name='sdk',
    version='0.1.0',
    description='Internal software development kit',
    long_description=readme,
    long_description_content_type='text/markdown',
    author='xxx team',
    author_email='admin@xxx.com',
    packages=packages,
    package_dir={'sdk': 'sdk'},
    python_requires=">=3.6",
    install_requires=requires,
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Natural Language :: English',
        'Topic :: Software Development :: Libraries',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3.6'
    ],
    tests_require=test_requirements
)

serverless: 1.31.1 serverless-python-requirements: 4.2.1

Any thoughts?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 23
  • Comments: 15 (4 by maintainers)

Commits related to this issue

Most upvoted comments

@bryantbiggs As mentioned here: https://github.com/pypa/pip/issues/562#issuecomment-19261632 please try to remove the -e before your requirements. Seems to be a long standing pip/py bug. šŸ˜› Confirmed it fixes it for me. Anyways, for something like this plugin, -e makes no sense because you wouldnā€™t ā€œeditā€ your files in a readonly environment (lambda). šŸ˜› @dschep after author confirmed, Iā€™d say safe to close this bug. Although, if you want, I/we can add a check to check for -eā€™s with eggā€™s and warn the user or something, eh? idk

+11
AndrewFarley on Sep 14, 2018

Thanks, ran into this too. Just a note, for some there is a reason to install in editable mode when developing and testing locally. This means we now have to have two different requirements files, one for deployment and one for development which seems unfortunate. If it was possible to strip it out this seems a really good solution.

+10
nshgraph on Feb 5, 2019

Hey, I ran into this. Thanks for reporting.

+8
blbradley on Jan 4, 2019

Hey guys, want a good laugh, this appears to be a pip/python bug. šŸ˜› Check it outā€¦

bash-4.2# python2.7 -m pip install --upgrade -t /var/task/ -r requirements.txt 
Obtaining measurements from git+https://github.com/jladan/package_demo.git#egg=measurements (from -r requirements.txt (line 3))
  Updating ./src/measurements clone
Collecting numpy (from measurements->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/c9/16/1134977cc35d2f72dbe80efa75a8e989ac21289f8e7e2c9005444cd17cd5/numpy-1.15.1-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, measurements
  Found existing installation: Measurements 0.5
    Uninstalling Measurements-0.5:
      Successfully uninstalled Measurements-0.5
  Running setup.py develop for measurements
    Complete output from command /usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/task/src/measurements/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-WMZs4g:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --home not recognized
    
    ----------------------------------------
  Rolling back uninstall of Measurements
Command "/usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/task/src/measurements/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-WMZs4g" failed with error code 1 in /var/task/src/measurements/

But if I remove the -t /var/task optionā€¦

bash-4.2# python2.7 -m pip install --upgrade -r requirements.txt 
Obtaining measurements from git+https://github.com/jladan/package_demo.git#egg=measurements (from -r requirements.txt (line 3))
  Updating ./src/measurements clone
Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib64/python2.7/site-packages (from measurements->-r requirements.txt (line 3)) (1.15.1)
Installing collected packages: measurements
  Found existing installation: Measurements 0.5
    Uninstalling Measurements-0.5:
      Successfully uninstalled Measurements-0.5
  Running setup.py develop for measurements
Successfully installed measurements

Thought itā€™d make you laugh a little like it did me. I did some extended testing, the -t /var/task/ is unnecessaryā€¦ so Iā€™ve removed it in an upcoming MR. Iā€™m doing a little more testing (running all tests) to make sure itā€™s fine and then Iā€™ll push it and MR it. So, fix incomingā€¦

+5
AndrewFarley on Sep 14, 2018
Read more comments on GitHub
← serverless-python-requirements: Poetry: missing .serverless/requirements.txt
serverless-python-requirements: Private repos failing to install →