Zappa: Error loading psycopg2 module on Python 3.6 (when on Lambda)

Context

Shiny new Zappa 0.41 and Python 3.6

Django==1.11 djangorestframework==3.6.2 psycopg2==2.6.1 zappa==0.41.0 …

Expected Behavior

It works fine locally, but on Lambda the following happens:

[1492873312435] File "/var/task/django_zappa_app.py", line 20, in get_django_wsgi
[1492873312435] return get_wsgi_application()
[1492873312435] File "/var/task/django/core/wsgi.py", line 13, in get_wsgi_application
[1492873312435] django.setup(set_prefix=False)
[1492873312435] File "/var/task/django/__init__.py", line 27, in setup
[1492873312435] apps.populate(settings.INSTALLED_APPS)
[1492873312435] File "/var/task/django/apps/registry.py", line 108, in populate
[1492873312435] app_config.import_models()
[1492873312435] File "/var/task/django/apps/config.py", line 202, in import_models
[1492873312436] self.models_module = import_module(models_module_name)
[1492873312436] File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
[1492873312436] return _bootstrap._gcd_import(name[level:], package, level)
[1492873312436] File "<frozen importlib._bootstrap>", line 978, in _gcd_import
[1492873312436] File "<frozen importlib._bootstrap>", line 961, in _find_and_load
[1492873312436] File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
[1492873312436] File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
[1492873312436] File "<frozen importlib._bootstrap_external>", line 678, in exec_module
[1492873312436] File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
[1492873312436] File "/var/task/django/contrib/auth/models.py", line 4, in <module>
[1492873312436] from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
[1492873312436] File "/var/task/django/contrib/auth/base_user.py", line 52, in <module>
[1492873312436] class AbstractBaseUser(models.Model):
[1492873312436] File "/var/task/django/db/models/base.py", line 124, in __new__
[1492873312436] new_class.add_to_class('_meta', Options(meta, app_label))
[1492873312436] File "/var/task/django/db/models/base.py", line 330, in add_to_class
[1492873312437] value.contribute_to_class(cls, name)
[1492873312437] File "/var/task/django/db/models/options.py", line 214, in contribute_to_class
[1492873312437] self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
[1492873312437] File "/var/task/django/db/__init__.py", line 33, in __getattr__
[1492873312437] return getattr(connections[DEFAULT_DB_ALIAS], item)
[1492873312437] File "/var/task/django/db/utils.py", line 211, in __getitem__
[1492873312437] backend = load_backend(db['ENGINE'])
[1492873312437] File "/var/task/django/db/utils.py", line 115, in load_backend
[1492873312437] return import_module('%s.base' % backend_name)
[1492873312437] File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
[1492873312437] return _bootstrap._gcd_import(name[level:], package, level)
[1492873312437] File "/var/task/django/db/backends/postgresql/base.py", line 25, in <module>
[1492873312437] raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
[1492873312437] django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dynamic module does not define module export function (PyInit__psycopg)
[1492873312437] During handling of the above exception, another exception occurred:
[1492873312437] Traceback (most recent call last):
[1492873312437] File "/var/runtime/awslambda/bootstrap.py", line 466, in <module>
[1492873312437] main()
[1492873312437] File "/var/runtime/awslambda/bootstrap.py", line 462, in main
[1492873312438] handle_event_request(request_handler, invokeid, event_body, context_objs, invoked_function_arn)
[1492873312438] File "/var/runtime/awslambda/bootstrap.py", line 248, in handle_event_request
[1492873312438] result = to_json(result)
[1492873312438] File "/var/runtime/awslambda/bootstrap.py", line 224, in to_json
[1492873312438] return json.dumps(obj, default=decimal_serializer)
[1492873312438] File "/var/lang/lib/python3.6/json/__init__.py", line 238, in dumps
[1492873312438] **kw).encode(obj)
[1492873312438] File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
[1492873312438] chunks = self.iterencode(o, _one_shot=True)
[1492873312438] File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
[1492873312438] return _iterencode(o, 0)
[1492873312438] File "/var/runtime/awslambda/bootstrap.py", line 42, in decimal_serializer
[1492873312439] raise TypeError(repr(o) + " is not JSON serializable")
[1492873312439] TypeError: <FrameSummary file /var/task/handler.py, line 514 in lambda_handler> is not JSON serializable
[1492873312598] Error loading psycopg2 module: dynamic module does not define module export function (PyInit__psycopg)
ImproperlyConfigured

Possible Fix

No idea yet, but before I dig further, I was wondering if someone had any pointers. I assume its something to do with pre-compiled C-extension packages not working on Python 3.6 yet.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 31 (17 by maintainers)

Most upvoted comments

I solved the issue by installing psycopg2-binary

+15

lukojewel on May 30, 2019

psycopg2-binary worked for me.

i was having this issue:

RuntimeError: populate() isn't reentrant

https://github.com/Miserlou/Zappa/issues/324

thank you!

oxbits on Aug 26, 2019

pipenv install psycopg2-binary also solved the issue for me, but I have 2 questions.

Is using psycopg2-binary suitable for a production environment? Seems like compiled code would be a good thing, but the docs at https://pypi.org/project/psycopg2-binary/ currently read “The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.”
If it is necessary for Django to work in a lambda using Zappa, should psycopg2-binary be added (or verified) as a dependency in the zappa init command. That would’ve saved me time tracking this down.

adamloving on Jul 23, 2019

I’m still getting No module named 'psycopg2._psycopg' with zappa==0.41.2 and psycopg2==2.7.1

blurrcat on May 10, 2017

I encountered the same problem today while running Zappa on my Mac. I executed the command pip install psycopg2-binary again, and everything worked fine.

nttrung143 on Dec 3, 2023

I ran across the "No module named ‘psycopg2’ problem, and resolved it by updating psycopg2 to 2.7.4 from 2.6.1.

isaac-jordan on May 20, 2018

[1505579116083] /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8: ImportError
Traceback (most recent call last):
  File "/var/task/handler.py", line 505, in lambda_handler
  return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 239, in lambda_handler
  handler = cls()
  File "/var/task/handler.py", line 131, in __init__
  self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/var/task/main.py", line 2, in <module>
  import psycopg2
  File "/var/task/psycopg2/__init__.py", line 50, in <module>
  from psycopg2._psycopg import (       # noqa
ImportError: /var/task/psycopg2/_psycopg.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

I have some problem like this…

RenderCoder on Sep 16, 2017

I think this may be the problem: in Python 3, there’s no match for psycopg2 in lambda-packages; later when we try to pull wheels from pip, psycopg2 is excluded because of this line:

 for installed_package_name in installed_packages_name_set:
     if installed_package_name not in lambda_packages:  # psycopg2 is in lambda_packages
         wheel_url = self.get_manylinux_wheel(installed_package_name)

So the actual version shipped to lambda is the one installed in the virtualenv.

blurrcat on May 10, 2017

Is your virtualenv definitely on python 3.6? Can you check if/what psycopg2 binary version is in the zip after zappa package? (the file name)

wagmiwiz on May 10, 2017