airflow: MYSQL fails to install due to incorrect GPG Key

Discussed in https://github.com/apache/airflow/discussions/36230

<div type='discussions-op-text'>

Originally posted by furquan39 December 14, 2023

Apache Airflow version

Other Airflow 2 version (please specify below)

If “Other Airflow 2 version” selected, which one?

2.4.1

What happened?

When attempting to rebuild an apache airflow image, we are getting failures during our builds when trying to run apt-get update.

The error we see: image

What you think should happen instead?

The airflow image should build successfully without any errors.

How to reproduce

Create a Dockerfile. Add the following lines to the Dockerfile:

`FROM apache/airflow:2.4.1-python3.8

ENV SSH_PASSWD “root:Docker!”

RUN pip3 install msal Authlib flask_oauthlib azure-storage-file-datalake apache-airflow-providers-databricks==3.1.0

USER root

RUN apt-get update
&& apt-get install -y --no-install-recommends
openssh-server
&& apt-get autoremove -yqq --purge
&& apt-get clean
&& rm -rf /var/lib/apt/lists/*`

Operating System

Debian 11

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

Seems like a similar issue to this: https://github.com/apache/airflow/issues/20911

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

</div>

About this issue

  • Original URL
  • State: closed
  • Created 7 months ago
  • Reactions: 10
  • Comments: 77 (48 by maintainers)

Commits related to this issue

Most upvoted comments

It looks as though the apt repository still has the old gpg key set: https://repo.mysql.com/apt/debian/conf/distributions

For us we couldn’t get the reference to the old key to go away. If it’s hardcoded in the repo itself, this would explain it.

Right now, for us what works is to remove the mysql repo from the sources in our docker image based on apache/airflow:2.6.3-python3.10

so

RUN rm /etc/apt/sources.list.d/mysql.list

Everyone who watched it - all the past released images of Airflow should have been patched with the new keys (and they should be also resilient to 2025 changes of the key - the mysql repository has been removed from apt repos (mysql is still installed in the images.

Closing the issue now - finally.

How about this:

RUN sudo gpg --recv-keys 3A79BD29

i think the new key version is: A8D3785C https://dev.mysql.com/doc/refman/8.2/en/checking-gpg-signature.html the thing is even if i set this before my apt-get update looks like it doesn’t override the default 467B942D3A79BD29 I’ve tried RUN apt-key adv --keyserver pgp.mit.edu --recv-keys A8D3785C
&& gpg --export “A8D3785C” > “/etc/apt/trusted.gpg.d/mysql.gpg”

Hi there, for those using Debian, and having the apt-key is deprecated. Manage keyring files in trusted.gpg.d instead. You should save a file in the trusted.gpg.d folder with the new key:

sudo wget https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 -O /etc/apt/trusted.gpg.d/mysql2023 sudo gpg --dearmor /etc/apt/trusted.gpg.d/mysql2023 sudo rm /etc/apt/trusted.gpg.d/mysql2023

Then update and upgrade as usual.

Edit: added the gpg --dearmor as it need to be binary for apt to read.

👋 Having same issue with airflow:2.6.2-python3.9

Fix is coming to all images

Temporary solution suggested by Jarek @potiuk for someone who extended airflow images, add this into your Docker file before run update/upgrade packages

RUN rm /etc/apt/sources.list.d/mysql.list

It looks as though the apt repository still has the old gpg key set: https://repo.mysql.com/apt/debian/conf/distributions

A quick workaround that should fix if anyone sees the problem (I can’t reproduce it - likely due to timezone). I will come with some better fix also for historical images but this should fix it for now (add it to your Dockerfile).

USER root 
RUN sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A8D3785C
RUN apt-get update ...

Also keyserver.ubuntu.com seems to be way better than the MIT one

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B7B3B788A8D3785C \ 
  && gpg --export "B7B3B788A8D3785C" > "/etc/apt/trusted.gpg.d/mysql.gpg"
RUN gpg --keyserver pgp.mit.edu --recv-keys B7B3B788A8D3785C \
         && gpg --export B7B3B788A8D3785C > "/etc/apt/trusted.gpg.d/mysql.gpg"

this is working now because they’ve updated the new key in mysql debian repo https://repo.mysql.com/apt/debian/conf/distributions

I also commented on all the issues opened in MySQL for that one, because I believe the policy of MySQL on sigining their repositories is deeply flawed and suggested them to change it.

Here is the comment I made:

Thanks for the fix. I think however the policy of Oracle/MySQL to have expiry date for your software is deeply flawed.

We had to manually fix all ~50 images we released in the past of our for Apache Airflow because of the expiry date.

Nobody else does it. Postgres, MariaDB, even MsSQL put no expiry date on the keys that are used to sign repos.

By putting expiry key on your apt repository you basically put an expiry date on your software and this expiry date gets shorter and shorter.

A good example of that are your own images that are affected. We had a user asking us for help in Airflow repo https://github.com/apache/airflow/issues/36231#issuecomment-1858419966

to help to fix the same issue with mysql:8.0.35-debian image of yours and we sent them to your support (as well, you should deal with your own problems).

This image has been released just 25 days ago. And due to the flawed policy of having an expiry date on your key, effectively lifetime of this image was 24 days. Not much. And likely you have a number of those images (similarly as what we had 50 of ours). Now I guess you need to retroactively rebuild/patch your images - which is something the flawed policy of yours made us to get 36 hours of scrambling and and answering support issues of our users (which we did despite our team is made of volunteers, not paid staff as is in the case of MySQL/Oracle).

We kinda lost faith in Oracle being a good steward of MySQL apt repos and we decided in Apache Airflow in accelerated discussion and (currently running) lazy consensus, to switch to MariaDB clients for all our future releases (including the 2.8.0 release that was actually delayed by at least 2 days because of this bug).

Lazy consensus thread here: https://lists.apache.org/list.html?dev@airflow.apache.org

I hope - for the sake of your users loosing days due to such issues, you will reconsider your policies around signing your APT repos.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B7B3B788A8D3785C

Where ?

No idea. Look in your image documentation. It’s mysql team that get us all into that mess, they shoudl fix their s*t and if they are poor in communicating where to look for help, then it’s bad thing for them.

thanks @souzagmu !

my update version: connexion to pgp.mit.edu was failing on my infra + removed apt-key deprecation message.

RUN gpg --keyserver keyserver.ubuntu.com --recv-keys B7B3B788A8D3785C \
 && gpg --export B7B3B788A8D3785C > "/etc/apt/trusted.gpg.d/mysql.gpg"

If you are not sure of the GPG signature or can’t find the mysql.list file, you can also just do this until a fix is coming out.

RUN apt update --allow-insecure-repositories

Yeah. That’s a quick fix if you need no mysql. I still cannot reproduce it even if I run it from scratch . So I think it really depends which cache you hit … I am afraid we have to wait until cache refreshes globally and then fix it.

Looks like Oracle did it again and changed their key.

All the Airflow 2.7* images should be patched now. Can someone who is using any airflow 2.7 double check it please? Also it’s a good opportunity to upgrade if you do not want to wait. I am not sure if I will be able to fix all past images of Airfllow - I cannot fully automate it because there are some subtle differences.

It works for me

All the Airflow 2.7* images should be patched now. Can someone who is using any airflow 2.7 double check it please? Also it’s a good opportunity to upgrade if you do not want to wait. I am not sure if I will be able to fix all past images of Airfllow - I cannot fully automate it because there are some subtle differences.

Yes. It will take a bit, but when I fix all images I will let everyone here know.

Cool - yeah. I kept on refreshing it… And started working on permanent fixes in the meantime. Small update: B7B3B788A8D3785C as full id (more secure)

we have a chance to get the same troubleshooting in the end of the 2024

Clarification in the end of October 2025

pub   rsa4096 2023-10-23 [SC] [expires: 2025-10-22]
          BCA43417C3B485DD128EC6D4B7B3B788A8D3785C
uid           MySQL Release Engineering <mysql-build@oss.oracle.com>
sub   rsa4096 2023-10-23 [E] [expires: 2025-10-22]

I think the problem might be on Oracle side, packages are signed by expired key 3A79BD29 and new GPG key doesn’t 3A79BD29, but according to the https://dev.mysql.com/doc/refman/8.0/en/checking-gpg-signature.html it should use new one:

[!NOTE]
The 3A79BD29 key expires on 2023-12-14. A new replacement key (A8D3785C) will sign upcoming MySQL 8.0.36 and higher packages. Both keys are installed by the MySQL repository setup packages released with MySQL 8.0.35, and both keys are also available at https://repo.mysql.com/.

I’ve tried to reproduce on main and this also happen to me on x86_64, ARM64 not affected as usual because it use MariaDB client libraries

docker build . \
    --build-arg DOCKER_CONTEXT_FILES=./docker-context-files \
    --build-arg INSTALL_MYSQL_CLIENT="true" \
    --build-arg INSTALL_MYSQL_CLIENT_TYPE=mysql \
    --build-arg INSTALL_MSSQL_CLIENT="true" \
    --build-arg INSTALL_POSTGRES_CLIENT="true" \
    --tag "airflow-image:dec-15"

Then I’ve tried to replace

https://github.com/apache/airflow/blob/01fd0d31b46682f4d700aaacf19cfe7a0fe9a057/scripts/docker/install_mysql.sh#L78

By add both keys

    export_key "467B942D3A79BD29" "mysql-2022"
    export_key "A8D3785C" "mysql-2023"

then I’ve got

7.158 Reading package lists...
7.428 W: GPG error: http://repo.mysql.com/apt/debian bookworm InRelease: The following signatures were invalid: EXPKEYSIG 467B942D3A79BD29 MySQL Release Engineering <mysql-build@oss.oracle.com>
7.428 E: The repository 'http://repo.mysql.com/apt/debian bookworm InRelease' is not signed.
------
Dockerfile:1506
--------------------
 1505 |     # That also protects against AUFS Docker backend problem where changing the executable bit required sync
 1506 | >>> RUN bash /scripts/docker/install_mysql.sh prod \
 1507 | >>>     && bash /scripts/docker/install_mssql.sh prod \
 1508 | >>>     && bash /scripts/docker/install_postgres.sh prod \
 1509 | >>>     && adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password \
 1510 | >>>            --quiet "airflow" --uid "${AIRFLOW_UID}" --gid "0" --home "${AIRFLOW_USER_HOME_DIR}" \
 1511 | >>> # Make Airflow files belong to the root group and are accessible. This is to accommodate the guidelines from
 1512 | >>> # OpenShift https://docs.openshift.com/enterprise/3.0/creating_images/guidelines.html
 1513 | >>>     && mkdir -pv "${AIRFLOW_HOME}" \
 1514 | >>>     && mkdir -pv "${AIRFLOW_HOME}/dags" \
 1515 | >>>     && mkdir -pv "${AIRFLOW_HOME}/logs" \
 1516 | >>>     && chown -R airflow:0 "${AIRFLOW_USER_HOME_DIR}" "${AIRFLOW_HOME}" \
 1517 | >>>     && chmod -R g+rw "${AIRFLOW_USER_HOME_DIR}" "${AIRFLOW_HOME}" \
 1518 | >>>     && find "${AIRFLOW_HOME}" -executable -print0 | xargs --null chmod g+x \
 1519 | >>>     && find "${AIRFLOW_USER_HOME_DIR}" -executable -print0 | xargs --null chmod g+x
 1520 |     
--------------------

When I keep only new one

    export_key "A8D3785C" "mysql"

then I’ve got

14.63 Reading package lists...
14.89 W: GPG error: http://repo.mysql.com/apt/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 467B942D3A79BD29
14.89 E: The repository 'http://repo.mysql.com/apt/debian bookworm InRelease' is not signed.
------
Dockerfile:1505
--------------------
 1504 |     # That also protects against AUFS Docker backend problem where changing the executable bit required sync
 1505 | >>> RUN bash /scripts/docker/install_mysql.sh prod \
 1506 | >>>     && bash /scripts/docker/install_mssql.sh prod \
 1507 | >>>     && bash /scripts/docker/install_postgres.sh prod \
 1508 | >>>     && adduser --gecos "First Last,RoomNumber,WorkPhone,HomePhone" --disabled-password \
 1509 | >>>            --quiet "airflow" --uid "${AIRFLOW_UID}" --gid "0" --home "${AIRFLOW_USER_HOME_DIR}" \
 1510 | >>> # Make Airflow files belong to the root group and are accessible. This is to accommodate the guidelines from
 1511 | >>> # OpenShift https://docs.openshift.com/enterprise/3.0/creating_images/guidelines.html
 1512 | >>>     && mkdir -pv "${AIRFLOW_HOME}" \
 1513 | >>>     && mkdir -pv "${AIRFLOW_HOME}/dags" \
 1514 | >>>     && mkdir -pv "${AIRFLOW_HOME}/logs" \
 1515 | >>>     && chown -R airflow:0 "${AIRFLOW_USER_HOME_DIR}" "${AIRFLOW_HOME}" \
 1516 | >>>     && chmod -R g+rw "${AIRFLOW_USER_HOME_DIR}" "${AIRFLOW_HOME}" \
 1517 | >>>     && find "${AIRFLOW_HOME}" -executable -print0 | xargs --null chmod g+x \
 1518 | >>>     && find "${AIRFLOW_USER_HOME_DIR}" -executable -print0 | xargs --null chmod g+x
 1519 |     
--------------------

Even latest provided by Oracle APT repo configurator (mysql-apt-config_0.8.28-1_all.deb) use RPM-GPG-KEY-mysql-2022 and not RPM-GPG-KEY-mysql-2023

@potiuk Maybe it is a good point to switch to MariaDB client libraries by default for x86_64? Because I guess we have a chance to get the same troubleshooting in the end of the 2024. And this is not first time when Oracle MySQL libraries have some problem when MariaDB not, e.g.:

Any - look it up in comments. I think most people in the discussions above (including Airlfow image) use keyserver.ubuntu.com and fallback to pgp.surf.nl - but look it up in general GPG docs and google, there are some keyserver pools and the like, It’s up to you to pick up the right servers following the distributed nature of GPG.

In additional MySQL 5.7 is End Of Life since end of the October 2023

Well. I can just repeat what I already wrote in https://github.com/apache/airflow/issues/36231#issuecomment-1858439236

No idea. Look in your image documentation. It’s mysql team that get us all into that mess, they shoudl fix their s*t and if they are poor in communicating where to look for help, then it’s bad thing for them.

They are aware about the issue, they were told that their bad policy made their own images unusable, and we told them that they need to regenerate their images. If they did not (you might try to pull them again and see if they did) then you need to ask them how to fix it. I don’t think anyone here is going to spend their time on fixing MySQL images in Airflow discussion.

We are all here volunteers and MySQL/Oracle has paid staff that should take care about fixing the problem they created in the first place.

any solution for mysql:5.7-debian?

my Dockerfile:

FROM mysql:5.7-debian

# http://mirrors.aliyun.com/debian
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29

RUN apt-get update && apt-get install -y --no-install-recommends \
        iputils-ping \
        vim-tiny \
        telnet \
        net-tools \
        pv \
        wget \
      && rm -rf /var/lib/apt/lists/*

COPY ./db-init.sql /docker-entrypoint-initdb.d/

EXPOSE 3306

@portoedu - let us know if your 2.5.3 images work

It works for me! Thanks @potiuk

2.1.* patched

@portoedu - let us know if your 2.5.3 images work

All the Airflow 2.7* images should be patched now. Can someone who is using any airflow 2.7 double check it please? Also it’s a good opportunity to upgrade if you do not want to wait. I am not sure if I will be able to fix all past images of Airfllow - I cannot fully automate it because there are some subtle differences.

We had a workaround in https://github.com/WordPress/openverse/pull/3527 for our Airflow image that was working, it’s now failing because of the patched images (heh) but the new images are building properly for us with this change reverted! Thanks so much for the quick action @potiuk!

All 2.6.* images should be fixed as well. Anyone using them please double check @renyu-capsule -> your image shoud be fixed.

Make sure to pull it not use the cache.

Other images might come later - I need to pick my guests now.

It looks like it’s your image and based on mysql - so this is not something we can help with - you need to look where and how the keys should be added for that image @nitin342 - ideally check with mysql support, the image is theirs not ours.

Getting another error after the fix:

 The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B7B3B788A8D3785C
Reading package lists...
W: GPG error: http://repo.mysql.com/apt/debian buster InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B7B3B788A8D3785C
E: The repository 'http://repo.mysql.com/apt/debian buster InRelease' is not signed.

PR for main and upcoming Airlfow 2.8.0 in https://github.com/apache/airflow/pull/36243

A quick workaround that should fix if anyone sees the problem (I can’t reproduce it - likely due to timezone). I will come with some better fix also for historical images but this should fix it for now (add it to your Dockerfile).

I added both the lines suggested in the issues and keep getting the error with this image apache/airflow:slim-2.7.3-python3.10.

RUN sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys A8D3785C
RUN sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29

Even this which is not recommended doesn’t pass the error

RUN apt update --allow-unauthenticated