velero: Restored postgresql tables missing after backup restoration in new cluster

What steps did you take and what happened:

  1. Deployed PostgreSQL in the my local cluster and created tables within the database.
  2. Created a Velero backup and It was completed successfully.
  3. Restored the Velero backup in a new cluster, including all pods, pvc, pv.
  4. Attempted to access the database in the new cluster to verify the presence of tables. But the expected tables were missing from the restored backup.

What did you expect to happen: After restoring the backup into a new cluster, the tables and data should be available for access in the new cluster.

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl exec -it -n postgres postgresql-0 -- psql -U postgres -d postgres
psql (16.2 (Debian 16.2-1.pgdg120+2)) 

Type "help" for help. 
postgres=# \d 
             List of relations 
Schema |    Name     |   Type   |  Owner    
--------+-------------+----------+---------- 
public | t_50000     | table    | postgres 
public | test        | table    | postgres 
public | test_id_seq | sequence | postgres 

(3 rows) 
  • velero backup create my-backup --include-namespaces postgres --default-volumes-to-fs-backup --snapshot-volumes --snapshot-move-data
Backup request "my-backup" submitted successfully. 

Run `velero backup describe my-backup` or `velero backup logs my-backup` for more details. 
  • velero backup describe my-backup
Name:         my-backup 
Namespace:    velero 
Labels:       velero.io/storage-location=velero-dev
Annotations:  velero.io/resource-timeout=10m0s 
              velero.io/source-cluster-k8s-gitversion=v1.27.3 
              velero.io/source-cluster-k8s-major-version=1 
              velero.io/source-cluster-k8s-minor-version=27 

Phase:  Completed 

Warnings: 
  Velero:     <none> 
  Cluster:    <none> 
  Namespaces: 
    postgres:   resource: /pods name: /postgresql-0 

Namespaces: 
  Included:  postgres 
  Excluded:  <none> 

Resources: 
  Included:        * 
  Excluded:        <none> 
  Cluster-scoped:  auto 

Label selector:  <none> 
Or label selector:  <none> 
Storage Location:  velero-dev

Velero-Native Snapshot PVs:  true 
Snapshot Move Data:          true 
Data Mover:                  velero 

TTL:  336h0m0s 

CSISnapshotTimeout:    10m0s 
ItemOperationTimeout:  4h0m0s 

Hooks:  <none> 

Backup Format Version:  1.1.0 

Started:    2024-03-05 20:17:33 +0330 +0330 
Completed:  2024-03-05 20:17:42 +0330 +0330 
Expiration:  2024-03-19 20:17:33 +0330 +0330 

Total items to be backed up:  16 
Items backed up:              16 

Velero-Native Snapshots: <none included> 
  • kubectl exec -it -n postgres postgresql-0 -- psql -U postgres -d postgres
psql (16.2 (Debian 16.2-1.pgdg120+2)) 

Type "help" for help. 

postgres=# /d 
postgres=# select * from test; 
ERROR:  relation "test" does not exist 

LINE 1: select * from test; 
                      ^ 
postgres=#  

Anything else you would like to add:

Environment:

  • Velero version (use 6.0.0 (Helm chart)):
  • Cloud provider or hardware configuration: Tested in local kind clusters
  • OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 18 (9 by maintainers)

Most upvoted comments

@Lyndon-Li “Probably, we need to add some checks to prevent these two backup flags coexist.” Actually, we probably don’t want that. default-volumes-to-fs-backup just specifies whether we’re using opt-in or opt-out. If you have a bunch of volumes you want to use fs-backup for (not compatible with CSI volumes), but a few CSI volumes you want to use datamover with, you’d use all 3 flags and then annotate the datamover volumes to opt in.

Point is that the flags are not contradictory – collectively, using all 3 means "by default, use fs-backup with individual volume opt-out, and for those opted-out volumes, snapshot the data and move to object storage via datamover.

@nknoii, If you can use CSI PVs, snapshot data mover backups can be done. They take CSI snapshot first and then backs up the data. But this method cannot be used for PVs that do not support CSI snapshots. For example, “local” type PVs (https://kubernetes.io/docs/concepts/storage/volumes/#local) do not support snapshots so you can’t use snapshot data mover method.

But you can use “File system backup” method (enabled by the option “–default-volumes-to-fs-backup”) but the only issue there, as @Lyndon-Li pointed out, is that backup is done from live PV. So your Postgres data may be backed up in inconsistent state, depending on DB activity. Note that you can use Velero hooks (https://velero.io/docs/v1.13/backup-hooks/) to quiesce data.

Hope this helps.

Probably, we need to add some checks to prevent these two backup flags coexist.

@nknoii From the command line you are using, you have specified both --default-volumes-to-fs-backup and --snapshot-move-data. --default-volumes-to-fs-backup activates the fs-backup method and --snapshot-move-data is for CSI snapshot data movement backup method. If they are both specified, as the current logic, fs-backup method takes preference.

However, fs-backup is not a consistent backup type, so your backup encountered data inconsistency.

Therefore, you need to remove --default-volumes-to-fs-backup so that you can get a crash consistent backup with CSI snapshot data movement backup method. For sure, CSI snapshot data movement backup requires your CSI driver supports CSI snapshot.