pravega: zookeeper logs show QuorumPeer Error closing file

Deployed Pravega-Operator image: pravega/pravega-operator:0.3.2 Pravega build used : 0.5.0-2189.cabe11b Zookeeper-operator image: pravega/zookeeper-operator:0.2.1 Ran small i/o workload (Total - 4 readers, 3 writers, 120 events/sec, ~ 100 KB/sec IO)

Environment details: PKS / K8 with medium cluster:

3 master nodes @ large.cpu (4 CPU, 4 GB Ram, 16 GB Disk)
10 worker nodes @ xlarge.cpu(8 cpu, 8 GB Ram, 32 GB Disk)
Tier-1 storage is from VSAN datastore
Tier-2 storage curved on NFS Client provisioner using Isilon as backend

See these errors in zk-0 , zk-1 and zk-2 even without restarting any zookeepers.

2019-04-04 09:59:37,732 [myid:1] - ERROR [ProcessThread(sid:1 cport:-1)::QuorumPeer@1564] - Error writing next dynamic config file to disk:
2019-04-04 09:59:37,736 [myid:1] - ERROR [LearnerHandler-/172.24.2.3:50172:QuorumPeer@1596] - Error closing file:
2019-04-04 09:59:37,737 [myid:1] - WARN  [LearnerHandler-/172.24.2.3:50172:QuorumPeer@1527] - Restarting Leader Election
2019-04-04 09:59:37,737 [myid:1] - INFO  [pravega-zk-0.pravega-zk-headless.default.svc.cluster.local/172.24.2.2:3888:QuorumCnxManager$Listener@892] - Leaving listener

zk_logs.zip

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17 (8 by maintainers)

Most upvoted comments

Yes… all pvc were deleted… $ kubectl get pvc

NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cache-pravega-pravega-segmentstore-0    Bound    pvc-603b878f-9ccd-11e9-aede-005056ba320d   30Gi       RWO            standard       26m
cache-pravega-pravega-segmentstore-1    Bound    pvc-88daf068-9ccd-11e9-aede-005056ba320d   30Gi       RWO            standard       25m
cache-pravega-pravega-segmentstore-2    Bound    pvc-9cd13017-9ccd-11e9-aede-005056ba320d   30Gi       RWO            standard       24m
data-pravega-zk-0                       Bound    pvc-e649cf07-9ccc-11e9-aede-005056ba320d   20Gi       RWO            standard       29m
data-pravega-zk-1                       Bound    pvc-00bb9f5f-9ccd-11e9-aede-005056ba320d   20Gi       RWO            standard       29m
data-pravega-zk-2                       Bound    pvc-1686f582-9ccd-11e9-aede-005056ba320d   20Gi       RWO            standard       28m
index-pravega-bookie-0                  Bound    pvc-60006dc1-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
index-pravega-bookie-1                  Bound    pvc-60096fd1-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
index-pravega-bookie-2                  Bound    pvc-601c4e9d-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
index-pravega-bookie-3                  Bound    pvc-602b82f0-9ccd-11e9-aede-005056ba320d   10Gi       RWO            standard       26m
journal-pravega-bookie-0                Bound    pvc-5ffc7aed-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
journal-pravega-bookie-1                Bound    pvc-600dd9b4-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
journal-pravega-bookie-2                Bound    pvc-6013e6a4-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
journal-pravega-bookie-3                Bound    pvc-602e30fe-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-0                 Bound    pvc-5ffe34d0-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-1                 Bound    pvc-600563c9-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-2                 Bound    pvc-60178a4e-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
ledger-pravega-bookie-3                 Bound    pvc-60284144-9ccd-11e9-aede-005056ba320d   100Gi      RWO            standard       26m
pravega-tier2                           Bound    pvc-575fabf9-9ccd-11e9-aede-005056ba320d   2000Gi     RWX            nfs            26m
pvc-shaka-zulu-nfs-client-provisioner   Bound    pv-shaka-zulu-nfs-client-provisioner       10Mi       RWO                           27m

Raghuveera-Rao on Jul 2, 2019