postgres-operator: WAL-G backup broken since 1.13.0, works in 1.12.2

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? v1.13.0
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? EKS
  • Are you running Postgres Operator in production? yes
  • Type of issue? Bug report

We create logical backups (pg_dump) and WAL-G+basebackups for our clusters. We use a k8s service account which is bound to an IAM role for S3 access. postgres-operator is deployed using helm.

When running operator version 1.12.2 (and spilo 16:3.2-p3), both logicalbackup cronjob and WAL-G+basebackups work as intented. I validate the WAL backup using PGUSER=postgres envdir "/run/etc/wal-e.d/env" /scripts/postgres_backup.sh "/home/postgres/pgdata/pgroot/data"

When I update the postgres-operator to 1.13.0 (and spilo to 16:3.3-p1), the logicalbackups still work, but the WAL+basebackup do not work anymore. When manually trying to create a basebackup with the same command, I get error:

create S3 storage: create new AWS session: configure session: assume role by ARN: InvalidParameter: 1 validation error(s) found.
- minimum field size of 2, AssumeRoleInput.RoleSessionName.

It seems to be an error specific to using a service account assuming an IAM role to access S3, specifically when running basebackup. Logicalbackup are able to put the pg_dump on S3 via the same authentication method

No other values were changed other than the spilo image, and helm chart version.

Let me know if you need additional information.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 17 (4 by maintainers)

Most upvoted comments

Yes, this is still happening in 3.3-p3 . The wal-g issue has been solved, merged, but no wal-g release with the fixed version yet. Is the only option for now is to downgrade to 3.2-p3?

yes, https://github.com/zalando/spilo/releases/tag/3.3-p3 is still affected by wal-g bug:

root@temporal-postgresql-0:/home/postgres# envdir etc/wal-e.d/env/ wal-g --version
wal-g version v3.0.3	3f88f3c	2024.08.08_17:53:40	PostgreSQL

root@temporal-postgresql-0:/home/postgres# envdir etc/wal-e.d/env/ wal-g backup-list
ERROR: 2025/01/15 10:45:28.592881 configure primary storage: configure storage with prefix "s3://postgres-backup/spilo/temporal-postgresql/dbc7de61-0d3d-410a-9dcf-0dba2d0a86dc/wal/15": create S3 storage: create new AWS session: configure session: assume role by ARN: InvalidParameter: 1 validation error(s) found.
- minimum field size of 2, AssumeRoleInput.RoleSessionName.

Have the same issue

Operator image: ghcr.io/zalando/postgres-operator:v1.14.0 Spilo image: ghcr.io/zalando/spilo-17:4.0-p2