moto: @mock_s3 permanently breaks requests library

@yhvh A workaround for this for now is:

import requests
import responses
import moto
import unittest

class TestRequests(unittest.TestCase):
    @moto.mock_s3
    def test_requests(self):
        responses.add_passthru('https://')
        response = requests.get('https://www.google.com')
        self.assertEqual(response.status_code, 200)

unittest.main()

As of moto 1.3.5 this works:

    import responses

    responses.add_passthru('https://')
    responses.add_passthru('http://')

Closed with #1553

@terrycain I’m kinda okay with non-supported services passing through to AWS. It seems reasonable to me that if you @mock_ec2 it goes to Moto and if you don’t you get AWS, always (If I understand the decision we’re trying to make correctly). @spulec WDYT?

Installed from master, this still fails. We have calls to different services interleaved with calls to aws, can you confirm this minimal use case is supported?

import moto, requests

@moto.mock_s3
def test_foo():
    print requests.get('https://www.google.co.uk')

    @moto.mock_s3
    def test_foo():
>       r = requests.get('https://www.google.co.uk')

test_foo.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/api.py:72: in get
    return request('get', url, params=params, **kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/api.py:58: in request
    return session.request(method=method, url=url, **kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/sessions.py:508: in request
    resp = self.send(prep, **send_kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/sessions.py:618: in send
    r = adapter.send(request, **kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/responses.py:572: in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <responses.RequestsMock object at 0x7f52e2515fd0>, adapter = <requests.adapters.HTTPAdapter object at 0x7f52df88dd90>, request = <PreparedRequest [GET]>, kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}, match = None, resp_callback = None, error_msg = 'Connection refused: GET https://www.google.co.uk/'
response = ConnectionError(u'Connection refused: GET https://www.google.co.uk/',)

    def _on_request(self, adapter, request, **kwargs):
        match = self._find_match(request)
        resp_callback = self.response_callback
    
        if match is None:
            if request.url.startswith(self.passthru_prefixes):
                logger.info(
                    'request.allowed-passthru', extra={
                        'url': request.url,
                    })
                return _real_send(adapter, request)
    
            error_msg = 'Connection refused: {0} {1}'.format(
                request.method, request.url)
            response = ConnectionError(error_msg)
            response.request = request
    
            self._calls.add(request, response)
            response = resp_callback(response) if resp_callback else response
>           raise response
E           ConnectionError: Connection refused: GET https://www.google.co.uk/

../../../.virtualenvs/testmoto/lib/python2.7/site-packages/responses.py:543: ConnectionError

@g-io I have worked around the problem with the following monkeypatch:

# https://github.com/spulec/moto/issues/1026#issuecomment-343525897
# https://github.com/terrycain/moto/blob/0e451fc1d1954f4fae750563f9cb290276b2e5c2/moto/packages/responses/responses.py
from moto.packages.responses import responses

responses.mock = responses._default_mock = responses.RequestsMock(
    assert_all_requests_are_fired=False,
    pass_through=True
)
for __attr in (a for a in dir(responses._default_mock) if not a.startswith('_')):
    setattr(responses, __attr, getattr(responses._default_mock, __attr))

Note that as of the new version of botocore, it no longer uses responses, so likely the solution above will fail too, whenever moto is rewritten to follow the change in botocore.

Explicitly calling the following upon the end of the mocked portion also appears to fix the issue.

            httpretty.HTTPretty.disable()
            httpretty.HTTPretty.reset()

This issue is biting us as well - we’re unable to use AWS services hosted in LocalStack docker containers if moto mocks are used anywhere in our test suite. I tried @amagee 's workaround successfully and got past my initial problem which was preventing me from talking to docker at all. However now in botocore’s urllib connection pool, I’m running into an odd error with an isinstance check on a Timeout class of all things - is it possible that this is a mocked resource that isn’t being cleaned up either? Haven’t been able to piece it together yet.

I’m having this issue also. I’m pretty sure the problem occurs when you use more than one mock from moto, for example decorating a test with mock_s3 and mock_lambda. My guess is something is going on with disabling patching. This will fail:

    def _process_lamda(pfunc):
        zip_output = io.BytesIO()
        zip_file = zipfile.ZipFile(zip_output, 'w', zipfile.ZIP_DEFLATED)
        zip_file.writestr('lambda_function.zip', pfunc)
        zip_file.close()
        zip_output.seek(0)
        return zip_output.read()

    def get_test_zip_file():
        pfunc = """
    def lambda_handler(event, context):
        return event
    """
    return _process_lamda(pounce)

    @mock_lambda
    @mock_apigateway
    def test_lambda_invoke(self):
        conn = boto3.client('lambda', 'us-west-2')
        conn.create_function(
            FunctionName='testMotoFunction',
            Runtime='python2.7',
            Role='test-iam-role',
            Handler='lambda_function.handler',
            Code={
                'ZipFile': get_test_zip_file(),
            },
            Description='test lambda function',
            Timeout=3,
            MemorySize=128,
            Publish=True,
            )

    def test_request(self):
        import requests
        r = requests.get('https://www.google.com')
        self.assertEqual(r.status_code, 200)

Traceback (most recent call last):
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/app/tests/common_tests/test_aws_lambda.py", line 36, in test_request
    r = requests.get('https://www.google.com')
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/moto/packages/responses/responses.py", line 308, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/moto/packages/responses/responses.py", line 250, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused: GET https://www.google.com/

Both tests will pass when removing the @mock_apigateway decorator.