docker-airflow: Incorrect padding errors from Fernet encryption

Thanks to @PedramNavid , setting FERNET_KEY env variable works, but is a workaround, IMO. Keep in mind that Fernet key must be 32 url-safe base64-encoded bytes, so doing openssl rand -base64 32 should generate you safe valid fernet key.

I have v1.10.1 and airflow initdb for postgesql 11 failed too. I required:

--python
>>> from cryptography.fernet import Fernet
>>> fernet_key= Fernet.generate_key()
>>> print(fernet_key.decode())
somelongkeyval
then

–bash export FERNET_KEY=‘somelongkeyval’; airflow initdb;

1. Look at script/entry point.sh - you’ll see that a Python command is being run and exported as an environment variable to create the Fernet key
2. Copy the Python code and package it into an export statement with the env variable name FERNET_KEY
3. In config/airflow.cfg, the fernet key is being defined as $FERNET_KEY, so its meant to pull from whatever you set the env variable as
4. Whenever you run a container, run that export statement - I do it through Docker exec in my Makefile

Note: this is definitely not the optimal way to do it, but I haven’t been able to get it to work at all in any other way. I tried putting it at the end of the entrypoint script, I tried running it as a command in the Dockerfile, but to no avail

HI

In my case, this work for me in bash: FERNET_KEY=$(python -c “from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)”) export FERNET_KEY=$FERNET_KEY

I was running into the same issue on Mac and found out it was my misunderstanding of how shell works:

The way I interact with the container was by docker exec -it {my-container-name} bash, which opens another process beside the original process that runs entrypoint.sh

Since it’s a separate process, it doesn’t have access to the environment variables exported by the entrypoint.sh. Therefore if I do echo $AIRFLOW__CORE__FERNET_KEY it returns null value, which is the reason of Incorrect padding errors

Now if I do source /entrypoint.sh before running airflow connection, the connection will be successfully added, but that means the fernet_key used will be different from the airflow.cfg. Therefore I guess the best way is to manually generate a fernet_key and pass it as an environment variable when you do the docker run, e.g. --env FERNET_KEY={my_key}

Hope this helps

Jealous! Also where are your logs stored? Can not for the life of me find them the way this image is set up

I will add this as another alternative if people want to still pull the image and use the SequentialExecutor for quick demos.

docker pull puckel/docker-airflow
YOUR_FERNET_KEY=$(openssl rand -base64 32)
docker run -d -e AIRFLOW__CORE__FERNET_KEY=$YOUR_FERNET_KEY -p 8080:8080 puckel/docker-airflow webserver

To update for folks who may find this for a google search, this was happening for me because my key was not base64 encoded (no = at the end). Changing AIRFLOW__CORE__FERNET_KEY= so it had a proper value allowed me to resetdb

I have the same issue while I was trying to add a new variable using the UI Admin tab -> Variables. The UI showed me an error message Could not create Fernet object: Incorrect padding

In my case, as @eduard-sukharev pointed out that Fernet key must be 32 url-safe base64-encoded bytes, but apparently I created one with illegal characters 😂

Solution is simply, I created a new one using openssl rand -base64 32, put it in our secret storage (we use Vault), and did the following

airflow@cdcf48897012:~$ /entrypoint.sh airflow resetdb

The entrypoint.sh contains the routine of reading the secrets from Vault, so it has to go first.

you’re welcome @KarthikRajashekaran 😃

Had the same issue, followed these steps to generate the Fernet key and replaced it in the airflow.cfg file. Worked out well!

running openssl rand -base64 32 helped me for updating from 1.9 to 1.10.1

Here are my logs related to this issue:

/entrypoint.sh: line 5: REDIS_HOST:=redis: command not found
/entrypoint.sh: line 6: REDIS_PORT:=6379: command not found
/entrypoint.sh: line 7: REDIS_PASSWORD:=: command not found
/entrypoint.sh: line 9: POSTGRES_HOST:=postgres: command not found
/entrypoint.sh: line 10: POSTGRES_PORT:=5432: command not found
/entrypoint.sh: line 11: POSTGRES_USER:=airflow: command not found
/entrypoint.sh: line 12: POSTGRES_PASSWORD:=airflow: command not found
/entrypoint.sh: line 13: POSTGRES_DB:=airflow: command not found 

The variable assignments are being executed as commands.