ceph-container: k8s (>= 1.9.4) deployment fails due to secret volumes mounted in read-only
From v1.9.4, secrets volumes are now mounted in read-only by default.
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v194 :
Changes secret, configMap, downwardAPI and projected volumes to mount read-only, instead of allowing applications to write data and then reverting it automatically. Until version 1.11, setting the feature gate ReadOnlyAPIDataVolumes=false will preserve the old behavior. (#58720, @joelsmith)
It is a problem because during the pod start, the scripts try to chown the /var/lib/ceph directory and the pods don’t start.
As a temporary workaround I’ve added “–feature-gates=ReadOnlyAPIDataVolumes=false” to the kubelet args and it works but It is definitely not recommended and the feature will only work until v1.9.11
Environment:
- Kubernetes version: v1.9.5
- Manifests used: https://github.com/ceph/ceph-container/tree/master/examples/kubernetes
- Container image: ceph/daemon:latest
- Ceph version: ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
$ kubectl get po -n ceph
NAME READY STATUS RESTARTS AGE
ceph-mds-7d8bb9c85b-ksjdl 0/1 CrashLoopBackOff 3 1m
ceph-mon-595c544bdf-6bdzk 0/1 Error 4 2m
ceph-mon-595c544bdf-hcdtc 0/1 CrashLoopBackOff 4 2m
ceph-mon-595c544bdf-skrlk 0/1 CrashLoopBackOff 3 2m
ceph-mon-check-7bdcf6f6c4-7wbg2 1/1 Running 0 1m
ceph-osd-5c6nd 0/1 Running 0 1m
ceph-osd-b92nn 0/1 Running 0 1m
ceph-osd-b96nf 0/1 Running 0 1m
ceph-osd-g5j9x 0/1 Running 0 1m
ceph-osd-kzppk 0/1 Running 1 1m
ceph-osd-mph8f 0/1 Running 0 1m
ceph-osd-p5klj 0/1 Running 0 1m
ceph-osd-wk7cp 0/1 Running 0 1m
ceph-osd-xc5mm 0/1 Running 0 1m
ceph-osd-xs9gf 0/1 Running 0 1m
$ kubectl logs ceph-mon-595c544bdf-skrlk -n ceph
Error from server (NotFound): pods "ceph-mon-595c544bdf-skrlk" not found
lcavajani@lcws:~/Workdir> kubectl logs ceph-mon-595c544bdf-6bdzk -n ceph
chown: changing ownership of '/var/lib/ceph/bootstrap-mds': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..2018_04_11_10_37_04.694425055': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..2018_04_11_10_37_04.694425055/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..2018_04_11_10_37_04.678163305': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..2018_04_11_10_37_04.678163305/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..2018_04_11_10_37_04.139056052': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..2018_04_11_10_37_04.139056052/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..2018_04_11_10_37_04.430911104': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..2018_04_11_10_37_04.430911104/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..2018_04_11_10_37_04.279864421': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..2018_04_11_10_37_04.279864421/ceph.keyring': Read-only file system
2018-04-11 10:38:59.992620 7f719d6a7180 -1 WARNING: the following dangerous and experimental features are enabled: btrfs
monmaptool: monmap file /etc/ceph/monmap-ceph
monmaptool: set fsid to 29fdb1d7-4393-40c3-baae-c29db46a2d53
monmaptool: writing epoch 0 to /etc/ceph/monmap-ceph (3 monitors)
bufferlist::write_file(/etc/ceph/monmap-ceph): failed to open file: (30) Read-only file system
monmaptool: error writing to '/etc/ceph/monmap-ceph': (30) Read-only file system
$ kubectl exec ceph-osd-xc5mm -n ceph – ls -la /var/lib/ceph
total 0
drwxrwxrwx 1 root root 174 Apr 11 10:37 .
drwxr-xr-x 1 root root 230 Apr 9 20:40 ..
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-mds
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-mgr
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-osd
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-rbd
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-rgw
drwxr-xr-x 1 ceph ceph 38 Apr 11 10:37 mds
drwxr-xr-x 1 ceph ceph 38 Apr 11 10:37 mgr
drwxr-xr-x 1 ceph ceph 38 Apr 11 10:37 mon
drwxrwxrwx 1 ceph ceph 0 Apr 11 10:37 osd
drwxr-xr-x 1 ceph ceph 46 Apr 11 10:37 radosgw
drwxr-xr-x 1 ceph ceph 84 Apr 11 10:40 tmp
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 17 (3 by maintainers)
Commits related to this issue
- daemon: move monmap to ephemeral storage dir In k8s 1.9.4 and later a fix for kubernetes/kubernetes#60814 makes mounted secrets and config maps read-only. This breaks the current ceph-container image... — committed to timn/ceph-container by timn 6 years ago
- daemon: move monmap to ephemeral storage dir In k8s 1.9.4 and later a fix for kubernetes/kubernetes#60814 makes mounted secrets and config maps read-only. This breaks the current ceph-container image... — committed to timn/ceph-container by timn 6 years ago
my work around:
The ReadOnlyAPIDataVolumes has been deprecated in kubernetes 1.10 and will be removed in 1.11. Having a bit of trouble enabling it in 1.10 but I am also new to kubernetes.
Using $ kubelet parameters add --feature-gates=ReadOnlyAPIDataVolumes=false seems to fail
kubelet parameters add --feature-gates=ReadOnlyAPIDataVolumes=false