k8up: Permission errors with nextcloud backups

Description

Hi k8up team!

First of all, thanks again for continuing to maintain this awesome project. 😃

I’m currently using 3 PVCs for hosting nextcloud on k3s, and I use restic via k8up to back them all up. One of the PVCs is not backed up properly, and it’s unfortunately the PVC where all my users’ files are 😅

I think this is because the directory has it’s “other” mode set to 0 (it’s 770) and is also owned by www-data, a non-root user that does not have a login shell. I’ve tried setting the security context to be www-data but that breaks things in even worse ways and nothing gets backed up when I do that. Is there something I should be doing differently? I’m unable to change the permissions/ownership of these files, as they’re created and maintained by nextcloud, and the www-data user is the admin user. See additional context for basic ls -hal output for the affected directories.

k8up still creates a restic snapshot for the data directory, but the snapshot has no files in that directory when checked with a local restic command.

Thanks for your time!

Additional Context

Permissions

Since I’m using k3s and their local path provisioner, I’m able to check the storage locally and see the permissions on the directory that is not backing up. The directory not being backed up is the data directory shown below

# pwd
/var/lib/rancher/k3s/storage/pvc-718e2220-f6ff-469c-8027-621ec4499111_nextcloud_nextcloud-datadir

# ls -hal data
drwxrwx--- 6 www-data www-data 4096 Dec 12 17:57 data/

# cd data && ls -hal
total 28
drwxrwx--- 6 www-data www-data 4096 Dec 12 17:57 ./
drwxrwxrwx 3 root     root     4096 Dec 12 17:51 ../
drwxr-xr-x 9 www-data www-data 4096 Dec 12 18:05 appdata_ocfqfwkgg2l9/
drwxr-xr-x 2 www-data www-data 4096 Dec 12 17:56 files_external/
-rw-r--r-- 1 www-data www-data  542 Dec 12 17:51 .htaccess
-rw-r--r-- 1 www-data www-data    0 Dec 12 17:51 index.html
-rw-r--r-- 1 www-data www-data    0 Dec 12 17:51 .ocdata
drwxr-xr-x 5 www-data www-data 4096 Dec 12 17:57 pizza_friend/

Logs

These are the errors in my backup pod from kubectl log:

1.6709268749080312e+09  INFO    k8up.restic.restic.backup       starting backup
1.670926874908162e+09   INFO    k8up.restic.restic.backup       starting backup for folder      {"foldername": "nextcloud-datadir"}
1.6709268749082112e+09  INFO    k8up.restic.restic.backup.command       restic command  {"path": "/usr/local/bin/restic", "args": ["backup", "--option", "", "--host", "nextcloud", "--json", "/data/nextcloud-datadir"]}
1.6709268749082484e+09  INFO    k8up.restic.restic.backup.command       Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
1.6709268815272708e+09  ERROR   k8up.restic.restic.backup.progress      /data/nextcloud-datadir/data during scan        {"error": "error occurred during backup"}
github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out
        /home/runner/work/k8up/k8up/restic/logging/logging.go:156
github.com/k8up-io/k8up/v2/restic/logging.writer.Write
        /home/runner/work/k8up/k8up/restic/logging/logging.go:103
io.copyBuffer
        /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429
io.Copy
        /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1
        /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407
os/exec.(*Cmd).Start.func1
        /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544
1.670926881527413e+09   ERROR   k8up.restic.restic.backup.progress      /data/nextcloud-datadir/data during archival    {"error": "error occurred during backup"}
github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out
        /home/runner/work/k8up/k8up/restic/logging/logging.go:156
github.com/k8up-io/k8up/v2/restic/logging.writer.Write
        /home/runner/work/k8up/k8up/restic/logging/logging.go:103
io.copyBuffer
        /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429
io.Copy
        /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1
        /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407
os/exec.(*Cmd).Start.func1
        /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544

Expected Behavior

Expected that restic would backup directory despite permissions.

Steps To Reproduce

Deploy k8up via the helm chart on k3s.

Deploy this PVC resource (on k3s)

# Dynamic persistent volume claim for nexctcloud data (/var/www/html/data) to persist
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: nextcloud
  name: nextcloud-datadir
  annotations:
    k8up.io/backup: "true"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Deploy Nextcloud

I deployed nextcloud using their official helm chart, and if you’d like, I can link my values.yaml, but the only relevant part is that I’m using an existing PVC claim, which is shown above. The version of the chart I’m using 3.3.3 and the version of nextcloud is 25.0.2.

Deploy Backup resource

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: test-backup-nextcloud-b2
  namespace: nextcloud
spec:
  failedJobsHistoryLimit: 10
  successfulJobsHistoryLimit: 10
  backend:
    repoPasswordSecretRef:
      name: k8up-restic-b2-repo-pw
      key: password
    b2:
      bucket: k8up-backups
      accountIDSecretRef:
        name: k8up-b2-creds
        key: application-key-id
      accountKeySecretRef:
        name: k8up-b2-creds
        key: application-key

Version of K8up

helm chart version: 3.0.2

Version of Kubernetes

v1.25.4+k3s1

Distribution of Kubernetes

k3s

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (22 by maintainers)

Most upvoted comments

Hi @jessebot

Great news! Thanks!

I would appreciate a bit of documentation about this mess. If you can spare the time I’m looking forward to a PR for that 😃

+1
Kidswiss on Dec 15, 2022

So I’ve just tested it through with B2 and their S3 API and… I wasn’t able to reproduce your issues unfortunately 😕

My steps:

  • created a new b2 key with all permissions
  • set the lifecycle to Keep only the last version
  • create fresh kind cluster
  • installed k8up via helm with defaults
  • created a mariadb deployment with pre-backup command and forced PVC backup
  • created a backup with random UID to force partial backup
  • created backup wit UID 0

I did not encounter a broken repository or the permission denied errors you’re currently having.

Here the commands:

Click me ```bash kind create cluster

helm repo add k8up-io https://k8up-io.github.io/k8up helm install k8up k8up-io/k8up

example with pre-backup and pvc backup

cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: labels: app: mariadb name: mariadb spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 2 selector: matchLabels: app: mariadb strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: mariadb annotations: k8up.io/backupcommand: mysqldump -uroot -psecure --all-databases spec: containers: - env: - name: MYSQL_ROOT_PASSWORD value: secure image: mariadb imagePullPolicy: Always name: mariadb ports: - containerPort: 3306 protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/lib/mysql name: mariadb dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - name: mariadb persistentVolumeClaim: claimName: mariadb

kind: PersistentVolumeClaim apiVersion: v1 metadata: annotations: k8up.io/backup: ‘true’ name: mariadb spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi EOF

cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: backup-credentials type: Opaque stringData: application-key-id: topsecret application-key: evenmoarsecret

apiVersion: v1 kind: Secret metadata: name: backup-repo type: Opaque stringData: password: asdf

apiVersion: k8up.io/v1 kind: Backup metadata: name: root-backup-nextcloud-b3 spec: podSecurityContext: runAsUser: 1337 #force partial backup failedJobsHistoryLimit: 10 successfulJobsHistoryLimit: 10 backend: repoPasswordSecretRef: name: backup-repo key: password s3: endpoint: https://s3.us-west-004.backblazeb2.com bucket: restic-test-k8up accessKeyIDSecretRef: name: backup-credentials key: application-key-id secretAccessKeySecretRef: name: backup-credentials key: application-key EOF

sleep 120

cat <<EOF | kubectl apply -f - apiVersion: k8up.io/v1 kind: Backup metadata: name: root-backup-nextcloud-b3 spec: podSecurityContext: runAsUser: 0 # working failedJobsHistoryLimit: 10 successfulJobsHistoryLimit: 10 backend: repoPasswordSecretRef: name: backup-repo key: password s3: endpoint: https://s3.us-west-004.backblazeb2.com bucket: restic-test-k8up accessKeyIDSecretRef: name: backup-credentials key: application-key-id secretAccessKeySecretRef: name: backup-credentials key: application-key EOF


some logs:
broken run:

1.6710910054898372e+09 INFO k8up Starting k8up… {“version”: “2.5.2”, “date”: “2022-10-26T15:32:22Z”, “commit”: “7b3cd59ffd75febca657e4db3fe7f8e789dfe109”, “go_os”: “linux”, “go_arch”: “amd64”, “go_version”: “go1.19.2”, “uid”: 1337, “gid”: 0} 1.6710910054905891e+09 INFO k8up.restic initializing 1.6710910054906611e+09 INFO k8up.restic setting up a signal handler 1.671091005491182e+09 INFO k8up.restic.restic using the following restic options {“options”: [“”]} 1.6710910054912121e+09 INFO k8up.restic.restic.RepoInit.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“init”, “–option”, “”]} 1.671091005491233e+09 INFO k8up.restic.restic.RepoInit.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710910069143622e+09 INFO k8up.restic.restic.unlock unlocking repository {“all”: false} 1.6710910069145362e+09 INFO k8up.restic.restic.unlock.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“unlock”, “–option”, “”]} 1.6710910069146202e+09 INFO k8up.restic.restic.unlock.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710910092935696e+09 INFO k8up.restic.restic.unlock.restic.stderr successfully removed locks 1.6710910093032126e+09 INFO k8up.restic.restic.snapshots getting list of snapshots 1.6710910093049035e+09 INFO k8up.restic.restic.snapshots.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“snapshots”, “–option”, “”, “–json”]} 1.6710910093055916e+09 INFO k8up.restic.restic.snapshots.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.671091013725819e+09 INFO k8up.restic.k8sClient listing all pods {“annotation”: “k8up.io/backupcommand”, “namespace”: “default”} 1.671091013753045e+09 INFO k8up.restic.k8sClient adding to backup list {“namespace”: “default”, “pod”: “mariadb-75f6846c85-29rsg”} 1.6710910137581148e+09 INFO k8up.restic.k8sExec executing command {“command”: “mysqldump, -uroot, -psecure, --all-databases”, “namespace”: “default”, “pod”: “mariadb-75f6846c85-29rsg”} 1.671091013759055e+09 INFO k8up.restic.restic.stdinBackup starting stdin backup {“filename”: “/default-mariadb”, “extension”: “”} 1.6710910137592638e+09 INFO k8up.restic.restic.stdinBackup.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“backup”, “–stdin”, “–stdin-filename”, “/default-mariadb”, “–host”, “default”, “–option”, “”, “–json”]} 1.6710910137593617e+09 INFO k8up.restic.restic.stdinBackup.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710910199643047e+09 INFO k8up.restic.restic.stdinBackup.progress backup finished {“new files”: 1, “changed files”: 0, “errors”: 0} 1.6710910199645727e+09 INFO k8up.restic.restic.stdinBackup.progress stats {“time”: 3.385511672, “bytes added”: 1196838, “bytes processed”: 5128593} 1.6710910199648566e+09 INFO k8up.restic.statsHandler.promStats sending prometheus stats {“url”: “http://127.0.0.1/”} 1.6710910199668348e+09 ERROR k8up.restic.restic prometheus send failed {“error”: “Post "http://127.0.0.1/metrics/job/restic_backup/instance/default\”: dial tcp 127.0.0.1:80: connect: connection refused"} github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats /home/runner/work/k8up/k8up/restic/cli/backup.go:95 github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:162 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710910201392481e+09 INFO k8up.restic backups of annotated jobs have finished successfully 1.6710910201393514e+09 INFO k8up.restic.restic.backup starting backup 1.6710910201395812e+09 INFO k8up.restic.restic.backup starting backup for folder {“foldername”: “mariadb”} 1.6710910201398492e+09 INFO k8up.restic.restic.backup.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“backup”, “–option”, “”, “–host”, “default”, “–json”, “/data/mariadb”]} 1.6710910201419113e+09 INFO k8up.restic.restic.backup.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.671091023925127e+09 INFO k8up.restic.restic.backup.progress progress of backup {“percentage”: “0.00%”} 1.6710910239263787e+09 ERROR k8up.restic.restic.backup.progress /data/mariadb/mysql during scan {“error”: “error occurred during backup”} github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:156 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.67109102392679e+09 ERROR k8up.restic.restic.backup.progress /data/mariadb/performance_schema during scan{“error”: “error occurred during backup”} github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:156 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710910239269178e+09 ERROR k8up.restic.restic.backup.progress /data/mariadb/sys during scan {“error”: “error occurred during backup”} github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:156 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710910246378922e+09 ERROR k8up.restic.restic.backup.progress /data/mariadb/mysql during archival {“error”: “error occurred during backup”} github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:156 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710910246389832e+09 ERROR k8up.restic.restic.backup.progress /data/mariadb/performance_schema during archival {“error”: “error occurred during backup”} github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:156 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.671091024642013e+09 ERROR k8up.restic.restic.backup.progress /data/mariadb/sys during archival {“error”: “error occurred during backup”} github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:156 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710910297652273e+09 INFO k8up.restic.restic.backup.progress backup finished {“new files”: 0, “changed files”: 0, “errors”: 6} 1.6710910297658556e+09 INFO k8up.restic.restic.backup.progress stats {“time”: 7.270300461, “bytes added”: 20409, “bytes processed”: 142615472} 1.6710910297660453e+09 INFO k8up.restic.statsHandler.promStats sending prometheus stats {“url”: “http://127.0.0.1/”} 1.6710910297673383e+09 ERROR k8up.restic.restic prometheus send failed {“error”: “Post "http://127.0.0.1/metrics/job/restic_backup/instance/default\”: dial tcp 127.0.0.1:80: connect: connection refused"} github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats /home/runner/work/k8up/k8up/restic/cli/backup.go:95 github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:162 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710910299530041e+09 INFO k8up.restic.restic.backup.progress restic output {“msg”: “Warning: at least one source file could not be read”} 1.6710910299567058e+09 INFO k8up.restic.restic.backup backup finished, sending snapshot list 1.671091029956774e+09 INFO k8up.restic.restic.snapshots getting list of snapshots 1.6710910299568179e+09 INFO k8up.restic.restic.snapshots.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“snapshots”, “–option”, “”, “–json”]} 1.6710910299568799e+09 INFO k8up.restic.restic.snapshots.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666}


working one (ignore the prom errors, haven't set it up locally :) ):

k logs backup-root-backup-nextcloud-b2-95j2w 1.671090911123626e+09 INFO k8up Starting k8up… {“version”: “2.5.2”, “date”: “2022-10-26T15:32:22Z”, “commit”: “7b3cd59ffd75febca657e4db3fe7f8e789dfe109”, “go_os”: “linux”, “go_arch”: “amd64”, “go_version”: “go1.19.2”, “uid”: 0, “gid”: 0} 1.67109091112444e+09 INFO k8up.restic initializing 1.671090911124518e+09 INFO k8up.restic setting up a signal handler 1.6710909111246688e+09 INFO k8up.restic.restic using the following restic options {“options”: [“”]} 1.671090911124788e+09 INFO k8up.restic.restic.RepoInit.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“init”, “–option”, “”]} 1.6710909111248682e+09 INFO k8up.restic.restic.RepoInit.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710909124638376e+09 INFO k8up.restic.restic.unlock unlocking repository {“all”: false} 1.6710909124642715e+09 INFO k8up.restic.restic.unlock.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“unlock”, “–option”, “”]} 1.6710909124643586e+09 INFO k8up.restic.restic.unlock.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710909150818992e+09 INFO k8up.restic.restic.unlock.restic.stderr successfully removed locks 1.6710909150843863e+09 INFO k8up.restic.restic.snapshots getting list of snapshots 1.6710909150846272e+09 INFO k8up.restic.restic.snapshots.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“snapshots”, “–option”, “”, “–json”]} 1.6710909150847232e+09 INFO k8up.restic.restic.snapshots.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710909195703094e+09 INFO k8up.restic.k8sClient listing all pods {“annotation”: “k8up.io/backupcommand”, “namespace”: “default”} 1.6710909195964541e+09 INFO k8up.restic.k8sClient adding to backup list {“namespace”: “default”, “pod”: “mariadb-75f6846c85-29rsg”} 1.671090919607463e+09 INFO k8up.restic.k8sExec executing command {“command”: “mysqldump, -uroot, -psecure, --all-databases”, “namespace”: “default”, “pod”: “mariadb-75f6846c85-29rsg”} 1.671090919609028e+09 INFO k8up.restic.restic.stdinBackup starting stdin backup {“filename”: “/default-mariadb”, “extension”: “”} 1.671090919609142e+09 INFO k8up.restic.restic.stdinBackup.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“backup”, “–json”, “–stdin”, “–stdin-filename”, “/default-mariadb”, “–option”, “”, “–host”, “default”]} 1.671090919609356e+09 INFO k8up.restic.restic.stdinBackup.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710909257196035e+09 INFO k8up.restic.restic.stdinBackup.progress backup finished {“new files”: 1, “changed files”: 0, “errors”: 0} 1.6710909257203126e+09 INFO k8up.restic.restic.stdinBackup.progress stats {“time”: 3.412426463, “bytes added”: 1196835, “bytes processed”: 5128593} 1.6710909257212725e+09 INFO k8up.restic.statsHandler.promStats sending prometheus stats {“url”: “http://127.0.0.1/”} 1.6710909257234204e+09 ERROR k8up.restic.restic prometheus send failed {“error”: “Post "http://127.0.0.1/metrics/job/restic_backup/instance/default\”: dial tcp 127.0.0.1:80: connect: connection refused"} github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats /home/runner/work/k8up/k8up/restic/cli/backup.go:95 github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:162 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.6710909260564318e+09 INFO k8up.restic backups of annotated jobs have finished successfully 1.6710909260565808e+09 INFO k8up.restic.restic.backup starting backup 1.6710909260576458e+09 INFO k8up.restic.restic.backup starting backup for folder {“foldername”: “mariadb”} 1.6710909260576868e+09 INFO k8up.restic.restic.backup.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“backup”, “–option”, “”, “–host”, “default”, “–json”, “/data/mariadb”]} 1.6710909260577228e+09 INFO k8up.restic.restic.backup.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666} 1.6710909297131133e+09 INFO k8up.restic.restic.backup.progress progress of backup {“percentage”: “0.00%”} 1.6710909319056516e+09 INFO k8up.restic.restic.backup.progress backup finished {“new files”: 194, “changed files”: 0, “errors”: 0} 1.6710909319070165e+09 INFO k8up.restic.restic.backup.progress stats {“time”: 3.50799872, “bytes added”: 545701, “bytes processed”: 154551900} 1.6710909319077353e+09 INFO k8up.restic.statsHandler.promStats sending prometheus stats {“url”: “http://127.0.0.1/”} 1.6710909319110754e+09 ERROR k8up.restic.restic prometheus send failed {“error”: “Post "http://127.0.0.1/metrics/job/restic_backup/instance/default\”: dial tcp 127.0.0.1:80: connect: connection refused"} github.com/k8up-io/k8up/v2/restic/cli.(*Restic).sendBackupStats /home/runner/work/k8up/k8up/restic/cli/backup.go:95 github.com/k8up-io/k8up/v2/restic/logging.(*BackupOutputParser).out /home/runner/work/k8up/k8up/restic/logging/logging.go:162 github.com/k8up-io/k8up/v2/restic/logging.writer.Write /home/runner/work/k8up/k8up/restic/logging/logging.go:103 io.copyBuffer /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:429 io.Copy /opt/hostedtoolcache/go/1.19.2/x64/src/io/io.go:386 os/exec.(*Cmd).writerDescriptor.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:407 os/exec.(*Cmd).Start.func1 /opt/hostedtoolcache/go/1.19.2/x64/src/os/exec/exec.go:544 1.671090932080978e+09 INFO k8up.restic.restic.backup backup finished, sending snapshot list 1.6710909320814269e+09 INFO k8up.restic.restic.snapshots getting list of snapshots 1.671090932081612e+09 INFO k8up.restic.restic.snapshots.command restic command {“path”: “/usr/local/bin/restic”, “args”: [“snapshots”, “–option”, “”, “–json”]} 1.67109093208169e+09 INFO k8up.restic.restic.snapshots.command Defining RESTIC_PROGRESS_FPS {“frequency”: 0.016666666666666666}

</details>

So I suspect somewhere in your configs there might be an error somewhere causing the permission denied. Maybe with the secrets? Is it possible that they drifted from what was initially applied?
+1
Kidswiss on Dec 15, 2022

I haven’t got a k8up dev env ready currently. So I only did basic tests with restic directly so far.

export RESTIC_REPOSITORY="s3:https://s3.us-west-004.backblazeb2.com/test-restic-problem"

restic init
created restic repository 93fdd3d5f1 at s3:https://s3.us-west-004.backblazeb2.com/test-restic-problem

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

restic backup /Volumes/VMs/baas/data/one/random/dir1/  
repository 93fdd3d5 opened (repository version 2) successfully, password is correct
created new cache in /Users/simonbeck/Library/Caches/restic
no parent snapshot found, will read all files

Files:         100 new,     0 changed,     0 unmodified
Dirs:            7 new,     0 changed,     0 unmodified
Added to the repository: 100.042 MiB (100.023 MiB stored)

processed 100 files, 100.000 MiB in 0:06
snapshot 7084b217 saved

restic snapshots
repository 93fdd3d5 opened (repository version 2) successfully, password is correct
ID        Time                 Host                Tags        Paths
-----------------------------------------------------------------------------------------------------
7084b217  2022-12-14 16:12:58  macbook-vshn.local              /Volumes/VMs/baas/data/one/random/dir1
-----------------------------------------------------------------------------------------------------
1 snapshots

EDIT: will try it with k8up tomorrow.

+1
Kidswiss on Dec 14, 2022

FWIW:

I just tried to create a restic repo with the b2 backend. And it restic was not able to create it properly… restic init failed for me.

Using their s3 endpoints though worked as expected as far s I can see.

+1
Kidswiss on Dec 14, 2022

The k8up config looks reasonable.

Looking forward to the results!

EDIT:

Just realized, that the s3 endpoint needs https:// prefix

    s3:
      endpoint: https://s3.eu-central-003.backblazeb2.com
+1
Kidswiss on Dec 14, 2022

Hi

According to your post you’re still using the B2 backend configuration and thus the broken restic b2 client. The restic docs state, that you should setup the restic configs as an S3 backend for B2 and generate the credentials:

Follow the documentation to generate S3-compatible access keys and then setup restic as described at Amazon S3. This is expected to work better than using the Backblaze B2 backend directly.

So the config should probably look like something like this:

export RESTIC_REPOSITORY="s3:b2-s3-endpoint-url"
export RESTIC_PASSWORD_FILE=/etc/restic-password
 export AWS_ACCESS_KEY_ID ="S3 compatible accesskey"
 export AWS_SECRET_ACCESS_KEY ="S3 compatible secret key"

I really suspect that the corruption is somehow produced by restic itself when using B2 backend… Maybe in combination with the partial and then a full working backup.

If it would at all help, I could set up a test b2 bucket for the k8up-io team to test things on their end?

I might have a bit of time this week to test things. They provider 10Gb for free AFAIK, so I should be able to do some tests on my own. Thanks for the suggestion though 😃

+1
Kidswiss on Dec 14, 2022

Thanks for the extensive explanation, that paints a better picture. And nothing you’ve described should be able to break the repos.

I just had a look at the restic docs, and they have a warning about using B2:

Due to issues with error handling in the current B2 library that restic uses, the recommended way to utilize Backblaze B2 is by using its S3-compatible API. Follow the documentation to generate S3-compatible access keys and then setup restic as described at Amazon S3. This is expected to work better than using the Backblaze B2 backend directly. Different from the B2 backend, restic’s S3 backend will only hide no longer necessary files. Thus, make sure to setup lifecycle rules to eventually delete hidden files.

https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2

It looks like the B2 implementation in restic is currently broken 😦

That could explain the weird things we’re seeing here…

So can you try that S3 compatibility layer for B2?

Just for completeness sake I answer the questions anyway:

What logs or data can I provide to help look further into this?

Running restic locally with their debug flag would probably shed more light on things.

Is there any restic cache to be cleared on the k8up side of things?

Nope, the cache is freshly loaded for every job.

how do I enable this debug mode that was mentioned previously?

You can set the envVar K8UP_DEBUG to true on the operator.

This is just a complicated mess because kubernetes clusters have so many moving pieces haha 😃

Whoever said that k8s is easy was probably drunk or so 😛

+1
Kidswiss on Dec 14, 2022

Hi @jessebot

Thanks for using K8up and reporting issues!

I’ve tried setting the security context to be www-data but that breaks things in even worse ways and nothing gets backed up when I do that.

Where did you set the security context? In Nextcloud or in the backup job?

One thing that usually works with finicky permissions is setting the backup podSecurityContext.runAsUser: 0. That forces the pod to be run as root.

+1
Kidswiss on Dec 13, 2022
Read more comments on GitHub
← k8ssandra-operator: v0.38.4: Error: services "k8ssandra-operator-webhook-service" already exists when deploying with helm
vue-tippy: Component in tooltip doesn't work →