ceph-container: Operation Not Permitted on Multi Node OSD

I am trying to get this working in a multi-node environment and I keep getting operation not permitted errors

I currently have one master node that runs a monitor. When I start an OSD on that node, it comes up fine. I then copied /etc/ceph and /var/lib/ceph to two more machines. When I start the OSD service on those nodes, I get this long output in my logs that seems to be triggered from an Operation not permitted error.

I am really new to ceph so I’m not really sure how to debug this though I did try fiddling with a few files and a lot of googling. Any idea where I’m getting it wrong?

--container="mesos-8aa60360-3d0a-4713-8fe2-60046f2af3c7-S0.aaf45dfd-a687-4d86-8e6c-0e3407979847" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slaves/8aa60360-3d0a-4713-8fe2-60046f2af3c7-S0/frameworks/8d0d7003-3072-4176-bf49-3d1551cfefb8-0000/executors/ceph_osd.3571d6e3-f933-11e6-a02d-024287699a2c/runs/aaf45dfd-a687-4d86-8e6c-0e3407979847" --stop_timeout="0ns"
--container="mesos-8aa60360-3d0a-4713-8fe2-60046f2af3c7-S0.aaf45dfd-a687-4d86-8e6c-0e3407979847" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slaves/8aa60360-3d0a-4713-8fe2-60046f2af3c7-S0/frameworks/8d0d7003-3072-4176-bf49-3d1551cfefb8-0000/executors/ceph_osd.3571d6e3-f933-11e6-a02d-024287699a2c/runs/aaf45dfd-a687-4d86-8e6c-0e3407979847" --stop_timeout="0ns"
Registered docker executor on medifor-tower0.sarnoff.internal
Starting task ceph_osd.3571d6e3-f933-11e6-a02d-024287699a2c
2017-02-22 19:15:27  /entrypoint.sh: static: does not generate config
ceph-1: /usr/bin/ceph-osd --cluster ceph -f -i 1 --osd-journal /var/lib/ceph/osd/ceph-1/journal -k /var/lib/ceph/osd/ceph-1/keyring
2017-02-22 19:15:27  /entrypoint.sh: SUCCESS
forego   | starting ceph-1.1 on port 31854
ceph-1.1 | starting osd.1 at - osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
ceph-1.1 | 2017-02-22 19:15:27.773422 7fde40a38a40 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
ceph-1.1 | 2017-02-22 19:15:27.792822 7fde40a38a40 -1 osd.1 0 log_to_monitors {default=true}
ceph-1.1 | 2017-02-22 19:15:27.805961 7fde40a38a40 -1  ** ERROR: osd init failed: (1) Operation not permitted
ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()' thread 7fde40a38a40 time 2017-02-22 19:15:27.806847
ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: 46: FAILED assert(m_workers.empty())
ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
ceph-1.1 |  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55b5d3251380]
ceph-1.1 |  2: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55b5d3337a71]
ceph-1.1 |  3: (CephContext::~CephContext()+0x428) [0x55b5d331f498]
ceph-1.1 |  4: (CephContext::put()+0xec) [0x55b5d331f78c]
ceph-1.1 |  5: (main()+0xd97) [0x55b5d2aa4f87]
ceph-1.1 |  6: (__libc_start_main()+0xf0) [0x7fde3e0b1830]
ceph-1.1 |  7: (_start()+0x29) [0x55b5d2b12dc9]
ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-1.1 | 2017-02-22 19:15:27.811335 7fde40a38a40 -1 /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()' thread 7fde40a38a40 time 2017-02-22 19:15:27.806847
ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: 46: FAILED assert(m_workers.empty())
ceph-1.1 | 
ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
ceph-1.1 |  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55b5d3251380]
ceph-1.1 |  2: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55b5d3337a71]
ceph-1.1 |  3: (CephContext::~CephContext()+0x428) [0x55b5d331f498]
ceph-1.1 |  4: (CephContext::put()+0xec) [0x55b5d331f78c]
ceph-1.1 |  5: (main()+0xd97) [0x55b5d2aa4f87]
ceph-1.1 |  6: (__libc_start_main()+0xf0) [0x7fde3e0b1830]
ceph-1.1 |  7: (_start()+0x29) [0x55b5d2b12dc9]
ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-1.1 | 
ceph-1.1 |    -98> 2017-02-22 19:15:27.773422 7fde40a38a40 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
ceph-1.1 |    -56> 2017-02-22 19:15:27.792822 7fde40a38a40 -1 osd.1 0 log_to_monitors {default=true}
ceph-1.1 |    -25> 2017-02-22 19:15:27.805961 7fde40a38a40 -1  ** ERROR: osd init failed: (1) Operation not permitted
ceph-1.1 |      0> 2017-02-22 19:15:27.811335 7fde40a38a40 -1 /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()' thread 7fde40a38a40 time 2017-02-22 19:15:27.806847
ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: 46: FAILED assert(m_workers.empty())
ceph-1.1 | 
ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
ceph-1.1 |  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55b5d3251380]
ceph-1.1 |  2: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55b5d3337a71]
ceph-1.1 |  3: (CephContext::~CephContext()+0x428) [0x55b5d331f498]
ceph-1.1 |  4: (CephContext::put()+0xec) [0x55b5d331f78c]
ceph-1.1 |  5: (main()+0xd97) [0x55b5d2aa4f87]
ceph-1.1 |  6: (__libc_start_main()+0xf0) [0x7fde3e0b1830]
ceph-1.1 |  7: (_start()+0x29) [0x55b5d2b12dc9]
ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-1.1 | 
ceph-1.1 | *** Caught signal (Aborted) **
ceph-1.1 |  in thread 7fde40a38a40 thread_name:ceph-osd
ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
ceph-1.1 |  1: (()+0x946e4e) [0x55b5d307fe4e]
ceph-1.1 |  2: (()+0x11390) [0x7fde3fad2390]
ceph-1.1 |  3: (gsignal()+0x38) [0x7fde3e0c6428]
ceph-1.1 |  4: (abort()+0x16a) [0x7fde3e0c802a]
ceph-1.1 |  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55b5d325156b]
ceph-1.1 |  6: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55b5d3337a71]
ceph-1.1 |  7: (CephContext::~CephContext()+0x428) [0x55b5d331f498]
ceph-1.1 |  8: (CephContext::put()+0xec) [0x55b5d331f78c]
ceph-1.1 |  9: (main()+0xd97) [0x55b5d2aa4f87]
ceph-1.1 |  10: (__libc_start_main()+0xf0) [0x7fde3e0b1830]
ceph-1.1 |  11: (_start()+0x29) [0x55b5d2b12dc9]
ceph-1.1 | 2017-02-22 19:15:27.817025 7fde40a38a40 -1 *** Caught signal (Aborted) **
ceph-1.1 |  in thread 7fde40a38a40 thread_name:ceph-osd
ceph-1.1 | 
ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
ceph-1.1 |  1: (()+0x946e4e) [0x55b5d307fe4e]
ceph-1.1 |  2: (()+0x11390) [0x7fde3fad2390]
ceph-1.1 |  3: (gsignal()+0x38) [0x7fde3e0c6428]
ceph-1.1 |  4: (abort()+0x16a) [0x7fde3e0c802a]
ceph-1.1 |  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55b5d325156b]
ceph-1.1 |  6: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55b5d3337a71]
ceph-1.1 |  7: (CephContext::~CephContext()+0x428) [0x55b5d331f498]
ceph-1.1 |  8: (CephContext::put()+0xec) [0x55b5d331f78c]
ceph-1.1 |  9: (main()+0xd97) [0x55b5d2aa4f87]
ceph-1.1 |  10: (__libc_start_main()+0xf0) [0x7fde3e0b1830]
ceph-1.1 |  11: (_start()+0x29) [0x55b5d2b12dc9]
ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-1.1 | 
ceph-1.1 |      0> 2017-02-22 19:15:27.817025 7fde40a38a40 -1 *** Caught signal (Aborted) **
ceph-1.1 |  in thread 7fde40a38a40 thread_name:ceph-osd
ceph-1.1 | 
ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
ceph-1.1 |  1: (()+0x946e4e) [0x55b5d307fe4e]
ceph-1.1 |  2: (()+0x11390) [0x7fde3fad2390]
ceph-1.1 |  3: (gsignal()+0x38) [0x7fde3e0c6428]
ceph-1.1 |  4: (abort()+0x16a) [0x7fde3e0c802a]
ceph-1.1 |  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55b5d325156b]
ceph-1.1 |  6: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55b5d3337a71]
ceph-1.1 |  7: (CephContext::~CephContext()+0x428) [0x55b5d331f498]
ceph-1.1 |  8: (CephContext::put()+0xec) [0x55b5d331f78c]
ceph-1.1 |  9: (main()+0xd97) [0x55b5d2aa4f87]
ceph-1.1 |  10: (__libc_start_main()+0xf0) [0x7fde3e0b1830]
ceph-1.1 |  11: (_start()+0x29) [0x55b5d2b12dc9]
ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-1.1 | 


About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 21 (10 by maintainers)

Most upvoted comments

same problem here with k8s. running ceph by example. works fine but not persistent. when changing the osd-ds.yaml from emptyDir: {} to hostPath: osd starts with error :

format_color_text  text_fields
2017-03-28T08:36:30.843953879Z 2017-03-28 08:36:30  /entrypoint.sh: k8s: config is stored as k8s secrets.
2017-03-28T08:36:30.844438649Z 2017-03-28 08:36:30  /entrypoint.sh: k8s: does not generate the admin key. Use Kubernetes secrets instead.
2017-03-28T08:36:30.848442198Z ceph-1: /usr/bin/ceph-osd --cluster ceph -f -i 1 --osd-journal /var/lib/ceph/osd/ceph-1//journal -k /var/lib/ceph/osd/ceph-1//keyring
2017-03-28T08:36:30.849060753Z 2017-03-28 08:36:30  /entrypoint.sh: SUCCESS
2017-03-28T08:36:30.850075006Z forego   | starting ceph-1.1 on port 5000
2017-03-28T08:36:30.858519082Z ceph-1.1 | starting osd.1 at - osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1//journal
2017-03-28T08:36:30.885129353Z ceph-1.1 | 2017-03-28 08:36:30.884878 7f171435aa40 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2017-03-28T08:36:30.896103405Z ceph-1.1 | 2017-03-28 08:36:30.895876 7f171435aa40 -1 osd.1 0 log_to_monitors {default=true}
2017-03-28T08:36:30.898615730Z ceph-1.1 | 2017-03-28 08:36:30.898447 7f171435aa40 -1  ** ERROR: osd init failed: (1) Operation not permitted
2017-03-28T08:36:30.900420373Z ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()' thread 7f171435aa40 time 2017-03-28 08:36:30.898737
2017-03-28T08:36:30.900433648Z ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: 46: FAILED assert(m_workers.empty())
2017-03-28T08:36:30.900439757Z ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
2017-03-28T08:36:30.900442975Z ceph-1.1 |  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x558176f29380]
2017-03-28T08:36:30.900445872Z ceph-1.1 |  2: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55817700fa71]
2017-03-28T08:36:30.900448768Z ceph-1.1 |  3: (CephContext::~CephContext()+0x428) [0x558176ff7498]
2017-03-28T08:36:30.900451559Z ceph-1.1 |  4: (CephContext::put()+0xec) [0x558176ff778c]
2017-03-28T08:36:30.900454174Z ceph-1.1 |  5: (main()+0xd97) [0x55817677cf87]
2017-03-28T08:36:30.900456657Z ceph-1.1 |  6: (__libc_start_main()+0xf0) [0x7f17119d2830]
2017-03-28T08:36:30.900459404Z ceph-1.1 |  7: (_start()+0x29) [0x5581767eadc9]
2017-03-28T08:36:30.900461841Z ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-03-28T08:36:30.900464902Z ceph-1.1 | 2017-03-28 08:36:30.900235 7f171435aa40 -1 /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: In function 'ceph::HeartbeatMap::~HeartbeatMap()' thread 7f171435aa40 time 2017-03-28 08:36:30.898737
2017-03-28T08:36:30.900467576Z ceph-1.1 | /tmp/buildd/ceph-11.2.0/src/common/HeartbeatMap.cc: 46: FAILED assert(m_workers.empty())
2017-03-28T08:36:30.900477184Z ceph-1.1 | 
2017-03-28T08:36:30.900479722Z ceph-1.1 |  ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
2017-03-28T08:36:30.900482051Z ceph-1.1 |  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x558176f29380]
2017-03-28T08:36:30.900484383Z ceph-1.1 |  2: (ceph::HeartbeatMap::~HeartbeatMap()+0xd1) [0x55817700fa71]
2017-03-28T08:36:30.900486872Z ceph-1.1 |  3: (CephContext::~CephContext()+0x428) [0x558176ff7498]
2017-03-28T08:36:30.900489152Z ceph-1.1 |  4: (CephContext::put()+0xec) [0x558176ff778c]
2017-03-28T08:36:30.900491468Z ceph-1.1 |  5: (main()+0xd97) [0x55817677cf87]
2017-03-28T08:36:30.900493745Z ceph-1.1 |  6: (__libc_start_main()+0xf0) [0x7f17119d2830]
2017-03-28T08:36:30.900496069Z ceph-1.1 |  7: (_start()+0x29) [0x5581767eadc9]
2017-03-28T08:36:30.900498610Z ceph-1.1 |  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
+2
J-Ha on Mar 28, 2017
Read more comments on GitHub
← ceph-container: k8s (>= 1.9.4) deployment fails due to secret volumes mounted in read-only
ceph-container: OSD unable to join CRUSH →