velero: Restored postgresql tables missing after backup restoration in new cluster
What steps did you take and what happened:
- Deployed PostgreSQL in the my local cluster and created tables within the database.
- Created a Velero backup and It was completed successfully.
- Restored the Velero backup in a new cluster, including all pods, pvc, pv.
- Attempted to access the database in the new cluster to verify the presence of tables. But the expected tables were missing from the restored backup.
What did you expect to happen: After restoring the backup into a new cluster, the tables and data should be available for access in the new cluster.
If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl exec -it -n postgres postgresql-0 -- psql -U postgres -d postgres
psql (16.2 (Debian 16.2-1.pgdg120+2))
Type "help" for help.
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------+----------+----------
public | t_50000 | table | postgres
public | test | table | postgres
public | test_id_seq | sequence | postgres
(3 rows)
velero backup create my-backup --include-namespaces postgres --default-volumes-to-fs-backup --snapshot-volumes --snapshot-move-data
Backup request "my-backup" submitted successfully.
Run `velero backup describe my-backup` or `velero backup logs my-backup` for more details.
velero backup describe my-backup
Name: my-backup
Namespace: velero
Labels: velero.io/storage-location=velero-dev
Annotations: velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.27.3
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=27
Phase: Completed
Warnings:
Velero: <none>
Cluster: <none>
Namespaces:
postgres: resource: /pods name: /postgresql-0
Namespaces:
Included: postgres
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Or label selector: <none>
Storage Location: velero-dev
Velero-Native Snapshot PVs: true
Snapshot Move Data: true
Data Mover: velero
TTL: 336h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2024-03-05 20:17:33 +0330 +0330
Completed: 2024-03-05 20:17:42 +0330 +0330
Expiration: 2024-03-19 20:17:33 +0330 +0330
Total items to be backed up: 16
Items backed up: 16
Velero-Native Snapshots: <none included>
kubectl exec -it -n postgres postgresql-0 -- psql -U postgres -d postgres
psql (16.2 (Debian 16.2-1.pgdg120+2))
Type "help" for help.
postgres=# /d
postgres=# select * from test;
ERROR: relation "test" does not exist
LINE 1: select * from test;
^
postgres=#
Anything else you would like to add:
Environment:
- Velero version (use
6.0.0 (Helm chart)): - Cloud provider or hardware configuration: Tested in local kind clusters
- OS (e.g. from
/etc/os-release):
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 18 (9 by maintainers)
@Lyndon-Li “Probably, we need to add some checks to prevent these two backup flags coexist.” Actually, we probably don’t want that. default-volumes-to-fs-backup just specifies whether we’re using opt-in or opt-out. If you have a bunch of volumes you want to use fs-backup for (not compatible with CSI volumes), but a few CSI volumes you want to use datamover with, you’d use all 3 flags and then annotate the datamover volumes to opt in.
Point is that the flags are not contradictory – collectively, using all 3 means "by default, use fs-backup with individual volume opt-out, and for those opted-out volumes, snapshot the data and move to object storage via datamover.
@nknoii, If you can use CSI PVs, snapshot data mover backups can be done. They take CSI snapshot first and then backs up the data. But this method cannot be used for PVs that do not support CSI snapshots. For example, “local” type PVs (https://kubernetes.io/docs/concepts/storage/volumes/#local) do not support snapshots so you can’t use snapshot data mover method.
But you can use “File system backup” method (enabled by the option “–default-volumes-to-fs-backup”) but the only issue there, as @Lyndon-Li pointed out, is that backup is done from live PV. So your Postgres data may be backed up in inconsistent state, depending on DB activity. Note that you can use Velero hooks (https://velero.io/docs/v1.13/backup-hooks/) to quiesce data.
Hope this helps.
Probably, we need to add some checks to prevent these two backup flags coexist.
@nknoii From the command line you are using, you have specified both
--default-volumes-to-fs-backupand--snapshot-move-data.--default-volumes-to-fs-backupactivates the fs-backup method and--snapshot-move-datais for CSI snapshot data movement backup method. If they are both specified, as the current logic, fs-backup method takes preference.However, fs-backup is not a consistent backup type, so your backup encountered data inconsistency.
Therefore, you need to remove
--default-volumes-to-fs-backupso that you can get a crash consistent backup with CSI snapshot data movement backup method. For sure, CSI snapshot data movement backup requires your CSI driver supports CSI snapshot.