VictoriaMetrics: Cluster version. Backup create/restore documentation is not clear

We are running cluster version 1.32.7 with two vmstorage nodes. Deployed with official helm chart

In order to implement data backups we went through the cluster backup guides. That mention to run vmbackup and vmrestore locally on each pod, but those binaries doesn’t exist in vmstorage-cluster docker image. Same as kill command doesn’t exist for scratch image base, so there is no way to stop vmstorage upon restore.

Should we build our own docker image with all required bits and pieces to make backup/restore available or we just missing something simple?

Any advice/documentation update is highly appreciated.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (4 by maintainers)

Most upvoted comments

Maybe it will be helpful for other people, too. Creating the backup was rather easy but restoring was a bit more difficult as it needs a lock on the data dir. First, I created a restore pod and copied the backup inside the pod:

vmrestore.yaml

apiVersion: v1
kind: Pod
metadata:
  name: vm-restore-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: server-volume-victoria-metrics-victoria-metrics-single-server-0
  containers:
    - name: vm-restore-container
      image: victoriametrics/vmrestore
      command: ["sleep"]
      args: ["infinity"]
      volumeMounts:
        - mountPath: "/storage"
          name: task-pv-storage

kubectl apply -f vmrestore.yaml

kubectl cp ~/Downloads/2021-07-06.tar.gz vm-restore-pod:/backup/2021-07-06.tar.gz

kubectl scale --replicas=0 statefulset/victoria-metrics-victoria-metrics-single-server

Shell into the restore pod and run:

tar -xzvf 2021-07-06.tar.gz
vmrestore-prod  -src=fs:///backup/2021-07-06 -storageDataPath=/storage/

kubectl scale --replicas=1 statefulset/victoria-metrics-victoria-metrics-single-server

It’s just quick and dirty written down, but maybe still helpful.

sj14 on Jul 6, 2021

Hi Guys, thanks for prompt reply. Sometime next week we are planning to migrate VM cluster between clouds. Thats when I’ll perform backup/restore exercise, will share my experience here.

kostyaplis on Mar 5, 2020