airflow: MYSQL fails to install due to incorrect GPG Key
Discussed in https://github.com/apache/airflow/discussions/36230
<div type='discussions-op-text'>Originally posted by furquan39 December 14, 2023
Apache Airflow version
Other Airflow 2 version (please specify below)
If “Other Airflow 2 version” selected, which one?
2.4.1
What happened?
When attempting to rebuild an apache airflow image, we are getting failures during our builds when trying to run apt-get update.
The error we see:
What you think should happen instead?
The airflow image should build successfully without any errors.
How to reproduce
Create a Dockerfile. Add the following lines to the Dockerfile:
`FROM apache/airflow:2.4.1-python3.8
ENV SSH_PASSWD “root:Docker!”
RUN pip3 install msal Authlib flask_oauthlib azure-storage-file-datalake apache-airflow-providers-databricks==3.1.0
USER root
RUN apt-get update
&& apt-get install -y --no-install-recommends
openssh-server
&& apt-get autoremove -yqq --purge
&& apt-get clean
&& rm -rf /var/lib/apt/lists/*`
Operating System
Debian 11
Versions of Apache Airflow Providers
No response
Deployment
Docker-Compose
Deployment details
No response
Anything else?
Seems like a similar issue to this: https://github.com/apache/airflow/issues/20911
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 7 months ago
- Reactions: 10
- Comments: 77 (48 by maintainers)
Commits related to this issue
- Temporarily switch to MariaDB client due to mysql repo signing issue We are temporarily switching to MariaDB client completely on our x86 builds to address a problem that mysql repo is now badly sign... — committed to potiuk/airflow by potiuk 7 months ago
- fix(pipeline) : Fix the build since Oracle again changed its signing key See https://github.com/apache/airflow/issues/36231 — committed to gip-inclusion/data-inclusion by vperron 7 months ago
- fix(pipeline) : Fix the build since Oracle again changed its signing key See https://github.com/apache/airflow/issues/36231 — committed to gip-inclusion/data-inclusion by vperron 7 months ago
- fix(pipeline) : Fix the build since Oracle again changed its signing key See https://github.com/apache/airflow/issues/36231 — committed to gip-inclusion/data-inclusion by vperron 7 months ago
- Change default MySQL client to MariaDB This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best practices for... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB (#36243) This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best pr... — committed to apache/airflow by potiuk 7 months ago
- Change default MySQL client to MariaDB (#36243) This PR is a response to pretty catastrophic issue caused by expiring key on MySQL repository on 14th of December. Oracle does not follow the best prac... — committed to apache/airflow by potiuk 7 months ago
- Fix fvt mysql installation issue Workaround for https://github.com/docker-library/mysql/issues/1016 as provided in https://github.com/apache/airflow/issues/36231. Signed-off-by: David Enyeart <enyea... — committed to denyeart/fabric-ca by denyeart 6 months ago
- patch: aptでmysqlの署名キー期限切れが発生しておりインストールでコケるのをパッチする https://github.com/apache/airflow/issues/36231 — committed to 1natsu172/forked-isucon13 by 1natsu172 6 months ago
- Merge pull request #1 from 1natsu172/patch-mysql patch: aptでmysqlの署名キー期限切れが発生しておりインストールでコケるのをパッチする https://github.com/apache/airflow/issues/36231 — committed to 1natsu172/forked-isucon13 by 1natsu172 6 months ago
- Revert "patch: aptでmysqlの署名キー期限切れが発生しておりインストールでコケるのをパッチする https://github.com/apache/airflow/issues/36231" — committed to 1natsu172/forked-isucon13 by 1natsu172 6 months ago
- Revert "patch: aptでmysqlの署名キー期限切れが発生しておりインストールでコケるのをパッチする https://github.com/apache/airflow/issues/36231" (#3) — committed to 1natsu172/forked-isucon13 by 1natsu172 6 months ago
For us we couldn’t get the reference to the old key to go away. If it’s hardcoded in the repo itself, this would explain it.
Right now, for us what works is to remove the mysql repo from the sources in our docker image based on
apache/airflow:2.6.3-python3.10
so
Everyone who watched it - all the past released images of Airflow should have been patched with the new keys (and they should be also resilient to 2025 changes of the key - the mysql repository has been removed from apt repos (mysql is still installed in the images.
Closing the issue now - finally.
i think the new key version is: A8D3785C https://dev.mysql.com/doc/refman/8.2/en/checking-gpg-signature.html the thing is even if i set this before my apt-get update looks like it doesn’t override the default 467B942D3A79BD29 I’ve tried RUN apt-key adv --keyserver pgp.mit.edu --recv-keys A8D3785C
&& gpg --export “A8D3785C” > “/etc/apt/trusted.gpg.d/mysql.gpg”
Hi there, for those using Debian, and having the apt-key is deprecated. Manage keyring files in trusted.gpg.d instead. You should save a file in the trusted.gpg.d folder with the new key:
sudo wget https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 -O /etc/apt/trusted.gpg.d/mysql2023
sudo gpg --dearmor /etc/apt/trusted.gpg.d/mysql2023
sudo rm /etc/apt/trusted.gpg.d/mysql2023
Then update and upgrade as usual.
Edit: added the gpg --dearmor as it need to be binary for apt to read.
Fix is coming to all images
Temporary solution suggested by Jarek @potiuk for someone who extended airflow images, add this into your Docker file before run update/upgrade packages
It looks as though the apt repository still has the old gpg key set: https://repo.mysql.com/apt/debian/conf/distributions
A quick workaround that should fix if anyone sees the problem (I can’t reproduce it - likely due to timezone). I will come with some better fix also for historical images but this should fix it for now (add it to your Dockerfile).
Also keyserver.ubuntu.com seems to be way better than the MIT one
this is working now because they’ve updated the new key in mysql debian repo https://repo.mysql.com/apt/debian/conf/distributions
I also commented on all the issues opened in MySQL for that one, because I believe the policy of MySQL on sigining their repositories is deeply flawed and suggested them to change it.
Here is the comment I made:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B7B3B788A8D3785C
No idea. Look in your image documentation. It’s mysql team that get us all into that mess, they shoudl fix their s*t and if they are poor in communicating where to look for help, then it’s bad thing for them.
thanks @souzagmu !
my update version: connexion to pgp.mit.edu was failing on my infra + removed apt-key deprecation message.
If you are not sure of the GPG signature or can’t find the
mysql.list
file, you can also just do this until a fix is coming out.Yeah. That’s a quick fix if you need no mysql. I still cannot reproduce it even if I run it from scratch . So I think it really depends which cache you hit … I am afraid we have to wait until cache refreshes globally and then fix it.
Looks like Oracle did it again and changed their key.
It works for me
All the Airflow 2.7* images should be patched now. Can someone who is using any airflow 2.7 double check it please? Also it’s a good opportunity to upgrade if you do not want to wait. I am not sure if I will be able to fix all past images of Airfllow - I cannot fully automate it because there are some subtle differences.
Yes. It will take a bit, but when I fix all images I will let everyone here know.
Cool - yeah. I kept on refreshing it… And started working on permanent fixes in the meantime. Small update:
B7B3B788A8D3785C
as full id (more secure)Clarification in the end of October 2025
I think the problem might be on Oracle side, packages are signed by expired key
3A79BD29
and new GPG key doesn’t3A79BD29
, but according to the https://dev.mysql.com/doc/refman/8.0/en/checking-gpg-signature.html it should use new one:I’ve tried to reproduce on main and this also happen to me on x86_64, ARM64 not affected as usual because it use MariaDB client libraries
Then I’ve tried to replace
https://github.com/apache/airflow/blob/01fd0d31b46682f4d700aaacf19cfe7a0fe9a057/scripts/docker/install_mysql.sh#L78
By add both keys
then I’ve got
When I keep only new one
then I’ve got
Even latest provided by Oracle APT repo configurator (mysql-apt-config_0.8.28-1_all.deb) use RPM-GPG-KEY-mysql-2022 and not RPM-GPG-KEY-mysql-2023
@potiuk Maybe it is a good point to switch to MariaDB client libraries by default for x86_64? Because I guess we have a chance to get the same troubleshooting in the end of the 2024. And this is not first time when Oracle MySQL libraries have some problem when MariaDB not, e.g.:
Any - look it up in comments. I think most people in the discussions above (including Airlfow image) use
keyserver.ubuntu.com
and fallback topgp.surf.nl
- but look it up in general GPG docs and google, there are some keyserver pools and the like, It’s up to you to pick up the right servers following the distributed nature of GPG.In additional MySQL 5.7 is End Of Life since end of the October 2023
Well. I can just repeat what I already wrote in https://github.com/apache/airflow/issues/36231#issuecomment-1858439236
They are aware about the issue, they were told that their bad policy made their own images unusable, and we told them that they need to regenerate their images. If they did not (you might try to pull them again and see if they did) then you need to ask them how to fix it. I don’t think anyone here is going to spend their time on fixing MySQL images in Airflow discussion.
We are all here volunteers and MySQL/Oracle has paid staff that should take care about fixing the problem they created in the first place.
any solution for mysql:5.7-debian?
my Dockerfile:
It works for me! Thanks @potiuk
2.1.* patched
@portoedu - let us know if your 2.5.3 images work
We had a workaround in https://github.com/WordPress/openverse/pull/3527 for our Airflow image that was working, it’s now failing because of the patched images (heh) but the new images are building properly for us with this change reverted! Thanks so much for the quick action @potiuk!
All 2.6.* images should be fixed as well. Anyone using them please double check @renyu-capsule -> your image shoud be fixed.
Make sure to pull it not use the cache.
Other images might come later - I need to pick my guests now.
It looks like it’s
your
image and based on mysql - so this is not something we can help with - you need to look where and how the keys should be added for that image @nitin342 - ideally check with mysql support, the image is theirs not ours.Getting another error after the fix:
PR for main and upcoming Airlfow 2.8.0 in https://github.com/apache/airflow/pull/36243
I added both the lines suggested in the issues and keep getting the error with this image
apache/airflow:slim-2.7.3-python3.10
.Even this which is not recommended doesn’t pass the error
Similar to https://github.com/apache/airflow/pull/20912