ceph-container: k8s (>= 1.9.4) deployment fails due to secret volumes mounted in read-only

From v1.9.4, secrets volumes are now mounted in read-only by default.

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#v194 :

Changes secret, configMap, downwardAPI and projected volumes to mount read-only, instead of allowing applications to write data and then reverting it automatically. Until version 1.11, setting the feature gate ReadOnlyAPIDataVolumes=false will preserve the old behavior. (#58720, @joelsmith)

It is a problem because during the pod start, the scripts try to chown the /var/lib/ceph directory and the pods don’t start.

As a temporary workaround I’ve added “–feature-gates=ReadOnlyAPIDataVolumes=false” to the kubelet args and it works but It is definitely not recommended and the feature will only work until v1.9.11

Environment:

$ kubectl get po -n ceph

NAME                              READY     STATUS             RESTARTS   AGE
ceph-mds-7d8bb9c85b-ksjdl         0/1       CrashLoopBackOff   3          1m
ceph-mon-595c544bdf-6bdzk         0/1       Error              4          2m
ceph-mon-595c544bdf-hcdtc         0/1       CrashLoopBackOff   4          2m
ceph-mon-595c544bdf-skrlk         0/1       CrashLoopBackOff   3          2m
ceph-mon-check-7bdcf6f6c4-7wbg2   1/1       Running            0          1m
ceph-osd-5c6nd                    0/1       Running            0          1m
ceph-osd-b92nn                    0/1       Running            0          1m
ceph-osd-b96nf                    0/1       Running            0          1m
ceph-osd-g5j9x                    0/1       Running            0          1m
ceph-osd-kzppk                    0/1       Running            1          1m
ceph-osd-mph8f                    0/1       Running            0          1m
ceph-osd-p5klj                    0/1       Running            0          1m
ceph-osd-wk7cp                    0/1       Running            0          1m
ceph-osd-xc5mm                    0/1       Running            0          1m
ceph-osd-xs9gf                    0/1       Running            0          1m

$ kubectl logs ceph-mon-595c544bdf-skrlk -n ceph

Error from server (NotFound): pods "ceph-mon-595c544bdf-skrlk" not found
lcavajani@lcws:~/Workdir> kubectl logs ceph-mon-595c544bdf-6bdzk -n ceph
chown: changing ownership of '/var/lib/ceph/bootstrap-mds': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..2018_04_11_10_37_04.694425055': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mds/..2018_04_11_10_37_04.694425055/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..2018_04_11_10_37_04.678163305': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-mgr/..2018_04_11_10_37_04.678163305/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..2018_04_11_10_37_04.139056052': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-osd/..2018_04_11_10_37_04.139056052/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..2018_04_11_10_37_04.430911104': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rbd/..2018_04_11_10_37_04.430911104/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..data': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..data/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/ceph.keyring': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..2018_04_11_10_37_04.279864421': Read-only file system
chown: changing ownership of '/var/lib/ceph/bootstrap-rgw/..2018_04_11_10_37_04.279864421/ceph.keyring': Read-only file system
2018-04-11 10:38:59.992620 7f719d6a7180 -1 WARNING: the following dangerous and experimental features are enabled: btrfs
monmaptool: monmap file /etc/ceph/monmap-ceph
monmaptool: set fsid to 29fdb1d7-4393-40c3-baae-c29db46a2d53
monmaptool: writing epoch 0 to /etc/ceph/monmap-ceph (3 monitors)
bufferlist::write_file(/etc/ceph/monmap-ceph): failed to open file: (30) Read-only file system
monmaptool: error writing to '/etc/ceph/monmap-ceph': (30) Read-only file system

$ kubectl exec ceph-osd-xc5mm -n ceph – ls -la /var/lib/ceph

total 0
drwxrwxrwx 1 root root 174 Apr 11 10:37 .
drwxr-xr-x 1 root root 230 Apr  9 20:40 ..
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-mds
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-mgr
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-osd
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-rbd
drwxrwxrwt 3 root root 100 Apr 11 10:37 bootstrap-rgw
drwxr-xr-x 1 ceph ceph  38 Apr 11 10:37 mds
drwxr-xr-x 1 ceph ceph  38 Apr 11 10:37 mgr
drwxr-xr-x 1 ceph ceph  38 Apr 11 10:37 mon
drwxrwxrwx 1 ceph ceph   0 Apr 11 10:37 osd
drwxr-xr-x 1 ceph ceph  46 Apr 11 10:37 radosgw
drwxr-xr-x 1 ceph ceph  84 Apr 11 10:40 tmp

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 1
  • Comments: 17 (3 by maintainers)

Commits related to this issue

Most upvoted comments

my work around:

volumeMounts:
  - name: ceph-conf
    mountPath: /tmp/ceph
lifecycle:
  postStart:
    exec:
      command:
        - "sh"
        - "-c"
        - cp /tmp/ceph/* /etc/ceph/

The ReadOnlyAPIDataVolumes has been deprecated in kubernetes 1.10 and will be removed in 1.11. Having a bit of trouble enabling it in 1.10 but I am also new to kubernetes.

Using $ kubelet parameters add --feature-gates=ReadOnlyAPIDataVolumes=false seems to fail

kubelet parameters add --feature-gates=ReadOnlyAPIDataVolumes=false