docker-airflow: libpq error

By adding below two library problem is resolved on mac but still facing the issue on Linux 'pip install psycopg2
&& pip install psycopg2-binary ’

I added libpq5 and the issue cleared up. Be mindful not to add it to the buildDeps section, as all of those libraries are removed at the end of the build.

Yes, just like @hbeadles is saying. Also in Airflow in your task log you see: “The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use “pip install psycopg2-binary” instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi.”

I can confirm this error. When building an image with the provided Dockerfile I see the following error when the image is run:

airflow-worker | [2019-04-08 21:18:19,697] {{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=1 airflow-worker | Traceback (most recent call last): airflow-worker | File “/usr/local/bin/airflow”, line 21, in <module> airflow-worker | from airflow import configuration airflow-worker | File “/usr/local/lib/python3.7/site-packages/airflow/init.py”, line 36, in <module> airflow-worker | from airflow import settings, configuration as conf airflow-worker | File “/usr/local/lib/python3.7/site-packages/airflow/settings.py”, line 266, in <module> airflow-worker | configure_orm() airflow-worker | File “/usr/local/lib/python3.7/site-packages/airflow/settings.py”, line 188, in configure_orm airflow-worker | engine = create_engine(SQL_ALCHEMY_CONN, **engine_args) airflow-worker | File “/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/init.py”, line 431, in create_engine airflow-worker | return strategy.create(*args, **kwargs) airflow-worker | File “/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py”, line 87, in create airflow-worker | dbapi = dialect_cls.dbapi(**dbapi_args) airflow-worker | File “/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py”, line 599, in dbapi airflow-worker | import psycopg2 airflow-worker | File “/usr/local/lib/python3.7/site-packages/psycopg2/init.py”, line 50, in <module> airflow-worker | from psycopg2._psycopg import ( # noqa airflow-worker | ImportError: libpq.so.5: cannot open shared object file: No such file or directory

@paulforan I can take a shot at answering. So from version 2.8 forward, psycopg2 no longer bundles binary packages. It bundles those separately in a new packages - psycopg2-binary. More information on the change can be found here: http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/. If you included postgres in that apache-airflow pip install line, it installs the newest version of postgres, (2.8.2), which does not have the binary packages included anymore. So to get around that, just remove postgres from there and install psycopg2-binary separately to get those dependencies included. More information can be found here – https://github.com/puckel/docker-airflow/pull/349

This worked for me: pip install psycopg2-binary redis>=3.2.0, which you can set with PYTHON_DEPS.

@paulforan I can take a shot at answering. So from version 2.8 forward, psycopg2 no longer bundles binary packages. It bundles those separately in a new packages - psycopg2-binary. More information on the change can be found here: http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/. If you included postgres in that apache-airflow pip install line, it installs the newest version of postgres, (2.8.2), which does not have the binary packages included anymore. So to get around that, just remove postgres from there and install psycopg2-binary separately to get those dependencies included. More information can be found here – #349

Yes This worked in my case, Thanks

Same issue. Faced it today.

[2019-07-08 08:29:15,214] {{settings.py:174}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=1 Traceback (most recent call last): File “/usr/local/bin/airflow”, line 21, in <module> from airflow import configuration File “/usr/local/lib/python3.6/site-packages/airflow/init.py”, line 36, in <module> from airflow import settings, configuration as conf File “/usr/local/lib/python3.6/site-packages/airflow/settings.py”, line 266, in <module> configure_orm() File “/usr/local/lib/python3.6/site-packages/airflow/settings.py”, line 188, in configure_orm engine = create_engine(SQL_ALCHEMY_CONN, **engine_args) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/init.py”, line 443, in create_engine return strategy.create(*args, **kwargs) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py”, line 87, in create dbapi = dialect_cls.dbapi(**dbapi_args) File “/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py”, line 599, in dbapi import psycopg2 File “/usr/local/lib/python3.6/site-packages/psycopg2/init.py”, line 50, in <module> from psycopg2._psycopg import ( # noqa ImportError: libpq.so.5: cannot open shared object file: No such file or directory

========== && pip install psycopg2==2.7.7
This fix helps me.

Probably another version conflict. In fact on my side I specified all the package versions:

    && pip install -U pip==19.0.1 setuptools==40.7.0 wheel==0.32.3 \
    && pip install pytz==2018.9  \
    && pip install pyOpenSSL==19.0.0 \
    && pip install ndg-httpsclient==0.5.1 \
    && pip install pyasn1==0.4.5 \
    && pip install psycopg2==2.7.7 \