charts: PgBouncer fails with Azure PostgreSQL

What is the bug?

I’m experimenting with the 8.5.2 chart version and when I’m installing with Pgbouncer feature disabled, everything works as usual but activating Pgbouncer flag on values.yaml, I get an error on the airflow-pgbouncer pod liveness probe:

Liveness probe failed: psql: error: ERROR: password authentication failed for user "[USERNAME]"

I’m using Azure for Postgres, and the username has an @ in the middle of the value but the log only shows the value before the @. I’m guessing this is why the auth is failing.

This is a continuation of issue #394

What version of the chart are you using?:

I am using version 8.5.2

What is your Kubernetes Version?:

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9", GitCommit:"1af681ff123a96cc50f4f6323a83c70a1daf6950", GitTreeState:"clean", BuildDate:"2021-05-25T17:37:16Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"clean", GoVersion:"go1.15.11"}

Please copy your custom Helm values file:

I’m using the default Pgbouncer config from the repo with the following change:

pgbouncer.serverSSL.mode: verify-ca

130823698-4e43b210-d835-4ef3-bc3b-493d942b044c

Thank you!

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 4
Comments: 16 (7 by maintainers)

Most upvoted comments

This issue is caused by Azure PostgreSQL NOT supporting md5 password authentication (only scram-sha-256), I have raised issue https://github.com/airflow-helm/charts/issues/419, to discuss adding a pgbouncer.authType value, which allows setting the auth_type config in pgbouncer.ini.

For now, if you are using the chart with a PostgreSQL that doesn’t support MD5 (like Azure PostgreSQL), you should disable pgbouncer by setting pgbouncer.enabled = false.

thesuperzapper on Sep 6, 2021

Therefore, once we add pgbouncer.authType (https://github.com/airflow-helm/charts/issues/419), we should update the docs to explain that Azure PostgreSQL needs these values:

pgbouncer:
  # WARNING: value does not exist as of chart `8.5.2`
  authType: scram-sha-256
  
  serverSSL:
    mode: verify-ca

thesuperzapper on Sep 6, 2021

I have also found that while using Azure PostgreSQL, you must set pgbouncer.serverSSL.mode to verify-ca. Even more strangely, you don’t have to provide a pgbouncer.serverSSL.caFile (which sets server_tls_ca_file) for it to work.

This is similar to what people raised in the PgBouncer issue tracker: https://github.com/pgbouncer/pgbouncer/issues/284

thesuperzapper on Sep 6, 2021

@george-zubrienko @minnieshi in my testing, the only change needed to use pgbouncer with Azure Postgress (on this chart) is to update templates/pgbouncer/_helpers/pgbouncer.tpl#L21 to auth_type = scram-sha-256.

This can be easily achieved by adding a new pgbouncer.authType value to the chart (see issue: https://github.com/airflow-helm/charts/issues/419).

NOTE: I have only been delaying doing this because I want the next minor release of the chart to support airflow 2.2.0 (it works already, but we need to add the new “deferable tasks” deployment (see issue: https://github.com/airflow-helm/charts/issues/424)

thesuperzapper on Nov 21, 2021

That is clear @george-zubrienko Thanks a lot. We don’t use TF, but I got the idea about the manual patch. (We use bash, helm cli, kubectl cli)

minnieshi on Nov 19, 2021

@minnieshi your best bet is to use the Azure PostgreSQL - Flexible Server (Preview), as this is a much more standard Postgress deployment, and works perfectly by default. (NOTE: it’s also MUCH more performant than the legacy Azure PostgreSQL)

If you really need to use the legacy Azure PostrgresSQL - Single Server, then you will need to either raise a PR for the chart that closes https://github.com/airflow-helm/charts/issues/419, or wait for me to do it. (Or you could just not use PgBouncer)

thesuperzapper on Nov 2, 2021