mariadb-operator: Error writing Galera Config. Permission Denied[Bug]

It looks like bumping up pvc to 500Mi worked. I think you can close the issue. However you may want to change the default size, or update the documentation for galera clusters appropriately.

“Can’t initialize timers” -> https://github.com/MariaDB/mariadb-docker/issues/434

TLDR;

• update docker version
• seccomp filter of the mariadb container isn’t sufficient for it to run

Thanks a lot @jonizen ! I’m glad that it finally works. I’m not a longhorn expert, do you happen to know why it requires a different set of permissions? Maybe writing to the PVC can only be performed by specific users? I would like to understand this as it’s not the first time it happens.

@anfechtung please give it a try, just to confirm so we can close this one.

I am no expert on longhorn either 😊 but know this request, talking about this. https://github.com/longhorn/longhorn/issues/1165

The request is for the possibility of have specific permissions for init. At that time longhorn did not respect security on text by do now.

And this reference to update the docs.

“Currently, Longhorn uses Bidirectional mount propagation, therefore it is allowed only in privileged containers.
Ref to https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation”

https://github.com/longhorn/longhorn/issues/3549

https://github.com/longhorn/longhorn/issues/3549#issuecomment-1046494049

Not sure this is to any help 😅

@grooverdan thanks for the information. I was able to get past that error by allowing the pod to run in an unconfined seccomp profile:

apiVersion: mariadb.mmontes.io/v1alpha1
kind: MariaDB
metadata:
  name: mariadb-galera
  namespace: cluster
spec:
  podSecurityContext:
    runAsUser: 0
  securityContext:
    seccompProfile:
      type: Unconfined

Next I encountered the following error:

DOCKER-SHELL1:/tmp/mariadb-operator$ k logs -n cluster mariadb-galera-0
Defaulted container "mariadb" out of: mariadb, agent, init (init)
2023-07-31 15:30:44+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
2023-07-31 15:30:44+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2023-07-31 15:30:44+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.0.2+maria~ubu2204 started.
2023-07-31 15:30:45+00:00 [Note] [Entrypoint]: Initializing database files
2023-07-31 15:30:45 0 [ERROR] mariadbd: Error writing file './ddl_recovery.log' (Errcode: 28 "No space left on device")
2023-07-31 15:30:45 0 [ERROR] Aborting
2023-07-31 15:30:45 0 [ERROR] mariadbd: Error on close of '(null)' (Errcode: 9 "Bad file descriptor")

I am working on debugging that issue now.

UPDATE:

Looks like there are issues with the data directory:

root@mariadb-galera-0:/# echo bob > /var/lib/mysql/bob
bash: echo: write error: No space left on device
root@mariadb-galera-0:/# 

I can write to the galera volume mount:

DOCKER-SHELL1:/tmp/mariadb-operator$ k exec -it -n cluster mariadb-galera-0 -- /bin/bash
root@mariadb-galera-0:/# cd /etc/mysql/mariadb.conf.d
root@mariadb-galera-0:/etc/mysql/mariadb.conf.d# ls
0-galera.cnf  1-bootstrap.cnf  lost+found
root@mariadb-galera-0:/etc/mysql/mariadb.conf.d# echo bob > bob
root@mariadb-galera-0:/etc/mysql/mariadb.conf.d# cat bob
bob
root@mariadb-galera-0:/etc/mysql/mariadb.conf.d# 

which implies that it is not an issue with the rook-ceph rbd volume.

The volume is, in fact, full:

root@mariadb-galera-0:/var/lib/mysql# df -h
Filesystem                    Size  Used Avail Use% Mounted on
overlay                        62G   18G   42G  30% /
tmpfs                          64M     0   64M   0% /dev
tmpfs                         6.4G     0  6.4G   0% /sys/fs/cgroup
/dev/mapper/vagrant--vg-root   62G   18G   42G  30% /etc/hosts
shm                            64M     0   64M   0% /dev/shm
/dev/rbd10                     44M   17K   43M   1% /etc/mysql/mariadb.conf.d
/dev/rbd9                      92M   90M     0 100% /var/lib/mysql
tmpfs                          13G   12K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                         6.4G     0  6.4G   0% /proc/acpi
tmpfs                         6.4G     0  6.4G   0% /proc/scsi
tmpfs                         6.4G     0  6.4G   0% /sys/firmware
root@mariadb-galera-0:/var/lib/mysql# 

root@mariadb-galera-0:/var/lib/mysql# ls -lah /var/lib/mysql
total 90M
drwxr-xr-x 3 mysql mysql 1.0K Jul 31 16:22 .
drwxr-xr-x 1 root  root  4.0K Jul  4 18:28 ..
-rw-rw---- 1 mysql mysql  16K Jul 31 15:23 aria_log.00000001
-rw-rw---- 1 mysql mysql   52 Jul 31 15:23 aria_log_control
-rw-r--r-- 1 root  root     0 Jul 31 16:22 bob
-rw-rw---- 1 mysql mysql  78M Jul 31 15:23 ib_logfile101
-rw-rw---- 1 mysql mysql  12M Jul 31 15:23 ibdata1
drwx------ 2 mysql mysql  12K Jul 31 15:22 lost+found
root@mariadb-galera-0:/var/lib/mysql# 

I am going to rebuild with a larger PVC. The 100Mi default may not be sufficient?

Did you recreate the galera cluster or just change the manifest and version for the init container? I had to recreate it, to make it work 😊

@anfechtunghttps://github.com/anfechtung interesting, I think your config file either got corrupted or is invalid, the init container seem to have worked. Could you provide the contents of /etc/mysql/mariadb.conf.d/0-galera.cnf?

@grooverdanhttps://github.com/grooverdan regarding the MariaDB container logs, does this logs ring a bell?