noobaa-core: Error in creating pv backend store

Environment info

• NooBaa Version: 5.2.6
• Platform: OpenShift 4.2

Actual behavior

Can’t create a deployment using pvpool from the gui or cli. It stays on initializing state. The agent pods are getting created and the PVC is mounted to the agent. It seems there is a connection issue between the agents and the core service.

Expected behavior

Creation of stores using a pvc backend

Steps to reproduce

install openshift 4.2 on baremetal with multitenancy enabled. install noobaa using the ocs operator on openshift 4.2

More information - Screenshots / Logs / Other output

CLI logs:

INFO[0000] ✅ Exists: NooBaa "noobaa"
INFO[0000] ✅ Exists: Service "noobaa-mgmt"
INFO[0000] ✅ Exists: Secret "noobaa-operator"
INFO[0000] ✈️  RPC: pool.create_hosts_pool() Request: {Name:rik1 IsManaged:true HostCount:1 HostConfig:{VolumeSize:10737418240}}
INFO[0000] ✅ RPC: pool.create_hosts_pool() Response OK: &{Op:res RequestID:1@https://::ffff:127.0.0.1:60942(1jijay6sxq) Took:483.784833997488 Error:<nil>}

noobaa core logs:

Oct-31 10:03:33.858 [WebServer/35] [ERROR] core.server.node_services.nodes_monitor:: got error in _get_agent_info: TimeoutError: operation timed out
    at afterTimeout (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:76:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
Oct-31 10:03:33.858 [WebServer/35] [ERROR] core.server.node_services.nodes_monitor:: got error in _get_agent_info: TimeoutError: operation timed out
    at afterTimeout (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:76:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
Oct-31 10:03:33.858 [WebServer/35] [ERROR] core.server.node_services.nodes_monitor:: got error in _get_agent_info: TimeoutError: operation timed out
    at afterTimeout (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:76:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
Oct-31 10:03:33.858 [WebServer/35]  [WARN] core.server.node_services.nodes_monitor:: _run_node: ERROR TimeoutError: operation timed out
    at afterTimeout (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:76:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10) node { _id: 5dbab0387b977b0023e6b1ae, peer_id: 5dbab0387b977b0023e6b1af, system: 5db837a07ea768002371cdeb, pool: 5dbab0217b977b0023e6b1ab, agent_config: 5dbab0217b977b0023e6b1ad, heartbeat: 1572515896913, name: 'a-node-has-no-name-k2ejd3kg1m9nkm0j4d', drives: [], latency_to_server: [], latency_of_disk_read: [], latency_of_disk_write: [], storage: { total: 0, free: 0, used: 0, alloc: 0, limit: 0 } }
Oct-31 10:03:33.858 [WebServer/35]  [WARN] core.server.node_services.nodes_monitor:: _run_node: ERROR TimeoutError: operation timed out
    at afterTimeout (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:76:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10) node { _id: 5dbab0397b977b0023e6b1b0, peer_id: 5dbab0397b977b0023e6b1b1, system: 5db837a07ea768002371cdeb, pool: 5dbab0217b977b0023e6b1ab, agent_config: 5dbab0217b977b0023e6b1ad, heartbeat: 1572515897530, name: 'a-node-has-no-name-k2ejd41m1m9nutsycb', drives: [], latency_to_server: [], latency_of_disk_read: [], latency_of_disk_write: [], storage: { total: 0, free: 0, used: 0, alloc: 0, limit: 0 } }
Oct-31 10:03:33.858 [WebServer/35]  [WARN] core.server.node_services.nodes_monitor:: _run_node: ERROR TimeoutError: operation timed out
    at afterTimeout (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:46:19)
    at Timeout.timeoutTimeout [as _onTimeout] (/root/node_modules/noobaa-core/node_modules/bluebird/js/release/timers.js:76:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10) node { _id: 5dbab0777b977b0023e6b1b5, peer_id: 5dbab0777b977b0023e6b1b6, system: 5db837a07ea768002371cdeb, pool: 5dbab0607b977b0023e6b1b2, agent_config: 5dbab0607b977b0023e6b1b4, heartbeat: 1572515959956, name: 'a-node-has-no-name-k2ejeg7o1magj8ahob', drives: [], latency_to_server: [], latency_of_disk_read: [], latency_of_disk_write: [], storage: { total: 0, free: 0, used: 0, alloc: 0, limit: 0 } }
Oct-31 10:03:34.566 [BGWorkers/32]   [LOG] CONSOLE:: TieringSpillbackWorker: start running