rook: OSD Processor -- bind unable to bind to IP on any port in range 6800-7300: (99) Cannot assign requested address

@travisn , still getting the error with 14.2.4, as in comment-563671883

For this container image and args:

  osd:
    Container ID: ____
    Image:         ___/ceph:v14.2.4-20190917
    Image ID:   ___
    Port:          <none>
    Host Port:     <none>
    Command:
      ceph-osd
    Args:
      --foreground
      --id
      1
      --conf
      /var/lib/docker/persistent/rook/osd1/kube-system.config
      --osd-data
      /var/lib/docker/persistent/rook/osd1
      --keyring
      /var/lib/docker/persistent/rook/osd1/keyring
      --cluster
      kube-system
      --osd-uuid
      ffda19d8-51ae-4b76-974f-6f0bdc64c1a4
      --default-log-to-file
      false
      --ms-learn-addr-from-peer=false

osd pod logs are:

2020-02-20 11:24:13.796 7f7faa09fdc0 -1 Falling back to public interface
2020-02-20 11:24:13.815 7f7faa09fdc0 -1  Processor -- bind unable to bind to v2:10.244.2.138:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2020-02-20 11:24:13.815 7f7faa09fdc0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds
2020-02-20 11:24:18.831 7f7faa09fdc0 -1  Processor -- bind unable to bind to v2:10.244.2.138:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2020-02-20 11:24:18.831 7f7faa09fdc0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds

@travisn It seems that the issue is not a port conflict, but the ceph-osd daemon picking the wrong IP address.

In my scenario above, the config file contained the right value for the pod IP:

+ kubectl -n rook-ceph exec rook-ceph-osd-2-86d8b7cbd9-brm6w -- cat /var/lib/rook/osd2/rook-ceph.config
[global]
...
public addr               = 10.34.0.9
cluster addr              = 10.34.0.9
...

but the OSD process, somewhere in pick_address(), decided to use 10.34.0.0 instead.

Also in pick_addresses.cc the code is looking at other config values (like public_network, for instance), which are not set in the config file passed to ceph-osd.

Perhaps sprinkling log statements in pick_address.cc and comparing the outputs of ceph-osd with hostNetwork: true (where it works) and hostNetwork: false could be of help …

It works! Thanks a lot, @leseb. I’m able to create an object store, and to read/write data from/to it.

I’m running now with the following configuration (diff from rook v1.0.2):

$ git diff
diff --git a/cluster/examples/kubernetes/ceph/cluster.yaml b/cluster/examples/kubernetes/ceph/cluster.yaml
index cb130e88..aacad270 100644
--- a/cluster/examples/kubernetes/ceph/cluster.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster.yaml
@@ -20,10 +20,10 @@ spec:
     # v12 is luminous, v13 is mimic, and v14 is nautilus.
     # RECOMMENDATION: In production, use a specific version tag instead of the general v14 flag, which pulls the latest release and could result in different
     # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
-    image: ceph/ceph:v14.2.1-20190430
+    image: ceph/daemon:latest-master
     # Whether to allow unsupported versions of Ceph. Currently luminous, mimic and nautilus are supported, with the recommendation to upgrade to nautilus.
     # Do not set to true in production.
-    allowUnsupported: false
+    allowUnsupported: true
   # The path on the host where configuration files will be persisted. Must be specified.
   # Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
   # In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
@@ -77,6 +77,13 @@ spec:
 # If no mgr annotations are set, prometheus scrape annotations will be set by default.
 #   mgr:
   resources:
+    mgr:
+      limits:
+        cpu: "1"
+        memory: "1024Mi"
+      requests:
+        cpu: "1"
+        memory: "1024Mi"
 # The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
 #    mgr:
 #      limits:
@@ -90,8 +97,8 @@ spec:
 #    osd:
   storage: # cluster level storage configuration and selection
     useAllNodes: true
-    useAllDevices: true
-    deviceFilter:
+    useAllDevices: false
+    deviceFilter: vdb
     location:
     config:
       # The default and recommended storeType is dynamically set to bluestore for devices and filestore for directories.
carlos@ubuntu-vm:~/src/rook/rook$ 

The resources snipper for mgr is to address a memory leak I reported earlier, nothing to do with this issue.

With this setup I see no errors in the logs, and Ceph looks healthy:

$ bash -x ./ceph-toolbox ceph status
++ kubectl -n rook-ceph get pods -lapp=rook-ceph-tools -o json
++ jq '.items[0].metadata.name'
++ tr -d '"'
+ pod=rook-ceph-tools-6488545f5b-dmf5q
+ '[' -n 'ceph status' ']'
+ exec kubectl -n rook-ceph exec rook-ceph-tools-6488545f5b-dmf5q -- ceph status
  cluster:
    id:     8051ed48-118a-458f-a438-2f9999241081
    health: HEALTH_WARN
            crush map has legacy tunables (require firefly, min is hammer)
 
  services:
    mon: 3 daemons, quorum a,b,c (age 27m)
    mgr: a(active, since 24m)
    osd: 4 osds: 4 up (since 25m), 4 in (since 25m)
    rgw: 1 daemon active (radosgw.gateway)
 
  data:
    pools:   6 pools, 600 pgs
    objects: 206 objects, 4.4 KiB
    usage:   4.0 GiB used, 392 GiB / 396 GiB avail
    pgs:     600 active+clean
 
$ bash -x ./ceph-toolbox ceph osd status
++ kubectl -n rook-ceph get pods -lapp=rook-ceph-tools -o json
++ jq '.items[0].metadata.name'
++ tr -d '"'
+ pod=rook-ceph-tools-6488545f5b-dmf5q
+ '[' -n 'ceph osd status' ']'
+ exec kubectl -n rook-ceph exec rook-ceph-tools-6488545f5b-dmf5q -- ceph osd status
+----+--------------+-------+-------+--------+---------+--------+---------+-----------+
| id |     host     |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | k8s-worker-1 | 1030M | 97.9G |    0   |     0   |    0   |     0   | exists,up |
| 1  | k8s-worker-0 | 1029M | 97.9G |    0   |     0   |    0   |     0   | exists,up |
| 2  | k8s-worker-3 | 1030M | 97.9G |    0   |     0   |    0   |     0   | exists,up |
| 3  | k8s-worker-2 | 1030M | 97.9G |    0   |     0   |    0   |     0   | exists,up |
+----+--------------+-------+-------+--------+---------+--------+---------+-----------+
$

Notice the health warning bit “crush map has legacy tunables (require firefly, min is hammer)” — anything to worry about?

@Blaine @leseb Any ideas on this issue with OSDs having port conflicts? #3189 also seems to be the same issue. It smells like we have a configuration issue and we’ll need to backport a fix to 1.0.

• We didn’t overhaul the osd config like the other daemons, but @Blaine can you think of config changes that might have affected the OSDs?
• @leseb Are there osd settings that might be affecting the bindings?

@gtorre you should always upgrade if a new version is available 😃. Right now, the only Ceph version that has the fix is Ceph master (Octopus dev), without that you don’t have any fix.

When ceph/ceph#28589 gets merged, ceph/daemon-base:latest-nautilus-devel will get the fix and eventually/hopefully Ceph 14.2.2 which is imminent.

In my deployed rook, ceph version is 14.2.4, but the problem is still there:

core@master3 ~ $ /opt/bin/kube/kubectl -n rook-ceph logs po/rook-ceph-osd-0-6b685b5478-nzsnt 2019-12-09 15:11:58.179 7f3478524dc0 -1 Falling back to public interface 2019-12-09 15:11:58.194 7f3478524dc0 -1 Processor – bind unable to bind to v2:10.244.70.5:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address 2019-12-09 15:11:58.194 7f3478524dc0 -1 Processor – bind was unable to bind. Trying again in 5 seconds 2019-12-09 15:12:03.209 7f3478524dc0 -1 Processor – bind unable to bind to v2:10.244.70.5:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address 2019-12-09 15:12:03.210 7f3478524dc0 -1 Processor – bind was unable to bind. Trying again in 5 seconds 2019-12-09 15:12:08.225 7f3478524dc0 -1 Processor – bind unable to bind to v2:10.244.70.5:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address 2019-12-09 15:12:08.225 7f3478524dc0 -1 Processor – bind was unable to bind after 3 attempts: (99) Cannot assign requested address

Set allowUnsupported: true in your CR.

and override this config option: ms_learn_addr_from_peer: false

Something like this?

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: rook-ceph
  name: rook-config-override
data:
  config: |
    [global]
    ms_learn_addr_from_peer = false

While I’m at it, shall I raise the debug level?

Perhaps sprinkling log statements in pick_address.cc and comparing the outputs of ceph-osd with hostNetwork: true (where it works) and hostNetwork: false could be of help …

I seem to be having the same problem as you only I have just one single OSD which refuses to start and as I verified with ceph osd dump also has an IP with x.x.x.0, the rest start just fine. However, I’m completely lost on how to start debugging this. The entry of participating has become enormous as I simply don’t know how to create a docker image with a custom Ceph build.