snowflake-connector-python: Release of boto3 1.10.2 breaks snowflake.connector, no workaround in AWS Lambda

Please answer these questions before submitting your issue. Thanks!

What version of Python are you using (python --version)? 3.7
What operating system and processor architecture are you using (python -c 'import platform; print(platform.platform())')? AWS Lambda

AWS released version 1.10.2 of boto3 yesterday, this is the version present in all Lambda execution environments, over which the end user has no control. https://pypi.org/project/boto3/#history

The boto3 release includes and update to botocore, which includes this commit: https://github.com/boto/botocore/commit/6ae424710109b1730cecdb1d3ae27a19a8454a58

The removal of botocore.vendored.requests breaks the import of snowflake.connector due to the dependence on botocore.vendored.requests here: https://github.com/snowflakedb/snowflake-connector-python/blob/master/network.py#L21.

This is a a very serious breaking issue, because all new Lambda executions come with a boto3/botocore environment provided by AWS, breaking existing deployed production code without it changing.

Suggested fix: if you need requests.adapters, add an explicit dependency rather than trying to pick it up transitively through botocore.

To reproduce:

import snowflake.connector

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 9
Comments: 16 (5 by maintainers)

Most upvoted comments

So… some very good news.

In stripping everything back to a minimal environment and busting all the CI caches etc. I was able to get a snowflake-connector working under lamba and boto3 using the following pinned versions:

In requirements.txt:

boto3==1.9.253
snowflake-connector-python==1.9.1

And then sam build, sam package, sam deploy.

I haven’t yet found a magic version that makes everything resolve with v2.0.2, but for the moment I don’t need that and 1.9.1 is just fine. (Note also I had no joy with 1.10.anything of boto3). I don’t know why all the things I thought I’d tried previously failed to get the same result, but building back up from here carefully I have been able to restore our production pipelines. I suspect a cached dependency somewhere which I’ve now been able to clear out.

So there’s still an issue to be resolved with the latest released version of the connector, but this is now not a ‘stop-the-line’ bug any more for me. Thank you for the prompt to recreate a minimal test environment, because without that I wouldn’t have been able to find the magic working combination. Hopefully this is also useful for anyone else hitting the same issue. On Tue, 29 Oct 2019 at 21:54, Shige Takeda notifications@github.com wrote:

Yes, it would be great if you provide your test case. I admit boto3 requirement is missing and should have been there, but I believe pip or requirement.txt can pin the version if required.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/snowflakedb/snowflake-connector-python/issues/221?email_source=notifications&email_token=AAAZ6EE7ZANVGNSUIZXAFGTQRCWJTA5CNFSM4JE6N5Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECSG2SQ#issuecomment-547646794, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAZ6EATOVDYI3JDUSNHAZDQRCWJTANCNFSM4JE6N5QQ.

acolyer on Oct 29, 2019

Any status on this? All our of data ingest pipelines have been broken in production since last Thursday. We really need you to put out a service release with the simplest fix possible (i.e., just just botocore.vendored.requests to requests and import requests directly) as quickly as possible. The turnaround time for something like that which is breaking customer code in production I would expect to be measured in hours, not days! Also a bit concerned that you let an issue you’ve known about since Aug 22 (#195) escalate to the point where such a production breakage happened. We’ll have to fork the connector and then update all of our pipeline packages if there is no progress on this immediately, which is something I would have preferred to have avoided.

If there’s some more major refactoring you want to do longer term, that can come in a follow-on release once the urgency level is reduced.

acolyer on Oct 29, 2019

Still seeing this

boto3==1.10.50
botocore==1.13.50
snowflake-connector-python==2.1.3

import snowflake.connector will break boto’s ability to communicate with AWS

sumew on Apr 3, 2020