velero: Restored pod with PVR failed to connect to the kube-apiserver
What steps did you take and what happened:
- Backup pod with PVC and restore post hooks
- Run restore
- Restore “Completes” but the post hook fails with error code 1 and the pasted error below
This example is running a script that runs a simple pg_restore command on an existing keycloak edb pod, but the keycloak-backup pod is completely cut off from the kube api. In order to run this pg_restore command, we need to exec into the running postgres pod which is why we need access to the kube api. I can create a new deployment that binds to the same PVC and manually run the script inside the newly deployed pod but the pod created by velero during the restore will never connect. I have noticed that the only difference in pod definition when recreating the pod is the removal of the restic init container which I suspect to be involved somehow. I have also found that once this issue is hit on a cluster, no amount of deleting and recreating the restored resources and the restore objects will change the outcome. Once I hit this issue, I always hit this issue and it prevents restore of this resource on this cluster.
Example of error:
time=“2024-02-02T17:43:26Z” level=info msg=“stdout: [INFO] Mode is set to restore, beginning restore process.\n” hookCommand=“[sh -c /keycloak/br_keycloak.sh restore int-oadp int-oadp]” hookContainer=keycloak-br hookName=“<from-annotation>” hookOnError=Fail hookPhase=post hookSource=annotation hookTimeout=“{5m0s}” hookType=exec logSource=“/remote-source/velero/app/pkg/podexec/pod_command_executor.go:173” pod=int-oadp/keycloak-backup-7cd87fb7f4-gwzkz restore=openshift-adp/ben-investigation-1time=“2024-02-02T17:43:26Z” level=info msg=“stderr: E0202 17:40:26.823349 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\”: dial tcp 172.30.0.1:443: i/o timeout\nE0202 17:40:56.824932 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\“: dial tcp 172.30.0.1:443: i/o timeout\nE0202 17:41:26.826748 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\”: dial tcp 172.30.0.1:443: i/o timeout\nE0202 17:41:56.827363 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\“: dial tcp 172.30.0.1:443: i/o timeout\nE0202 17:42:26.828493 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\”: dial tcp 172.30.0.1:443: i/o timeout\nE0202 17:42:56.829524 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\“: dial tcp 172.30.0.1:443: i/o timeout\nE0202 17:43:26.831342 13 memcache.go:238] couldn’t get current server API group list: Get "https://172.30.0.1:443/api?timeout=32s\”: dial tcp 172.30.0.1:443: i/o timeout\nUnable to connect to the server: dial tcp 172.30.0.1:443: i/o timeout\n" hookCommand=“[sh -c /keycloak/br_keycloak.sh restore int-oadp int-oadp]” hookContainer=keycloak-br hookName=“<from-annotation>” hookOnError=Fail hookPhase=post hookSource=annotation hookTimeout=“{5m0s}” hookType=exec logSource=“/remote-source/velero/app/pkg/podexec/pod_command_executor.go:174” pod=int-oadp/keycloak-backup-7cd87fb7f4-gwzkz restore=openshift-adp/ben-investigation-1time=“2024-02-02T17:43:26Z” level=error msg=“Error executing hook” error=“command terminated with exit code 1” hookPhase=post hookSource=annotation hookType=exec logSource=“/remote-source/velero/app/internal/hook/wait_exec_hook_handler.go:173” pod=int-oadp/keycloak-backup-7cd87fb7f4-gwzkz restore=openshift-adp/ben-investigation-1
Example of difference between pods with and without restic init container:
What did you expect to happen: Succesful completion of restore or at least a PartiallyFailed output instead of Completed.
The following information will help us better understand what’s going on:
If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help
bundle-2024-02-21-13-24-03.tar.gz
If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs deployment/velero -n velerovelero backup describe <backupname>orkubectl get backup/<backupname> -n velero -o yamlvelero backup logs <backupname>velero restore describe <restorename>orkubectl get restore/<restorename> -n velero -o yamlvelero restore logs <restorename>
Anything else you would like to add:
We are using velero via OADP 1.3. From what I can tell, this issue only happens when restoring pods with post restore hooks and I typically notice it when running OC commands (which is a kubectl wrapper for openshift). This is a very nasty bug that essentially bricks our ability to restore to a cluster that sees this problem. The problem is intermittent but once a cluster hits it once, it almost never works again.
Environment:
- Velero version (use
velero version): 1.12.1-OADP - Velero features (use
velero client config get features): <NOT SET> - Kubernetes version (use
kubectl version): v1.27.9+e36e183 - Kubernetes installer & version: OpenShift 4.14.10
- Cloud provider or hardware configuration: amd64
- OS (e.g. from
/etc/os-release): RHEL 8.8
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (5 by maintainers)
@shubham-pampattiwar @kaovilai Could you help also to take a look at this issue? I’m uncertain about whether this is OpenShift-specific. The filesystem uploader backs the restored workload’s volume data, but the restored workload cannot access the kube-apiserver.