tidb-operator: tikv not running

I using tidb-operator install a tidb on my three nodes k8s env, but the tikv not running. The result is as blew:

kubectl get pods -n tidb
NAME                              READY   STATUS             RESTARTS   AGE
demo-discovery-5468c7c556-5c624   1/1     Running            5          102m
demo-monitor-84446b7957-wrxlg     2/2     Running            0          102m
demo-monitor-configurator-v697r   1/1     Running            0          102m
demo-pd-0                         0/1     CrashLoopBackOff   5          102m
demo-pd-1                         1/1     Running            0          102m
demo-pd-2                         1/1     Running            5          102m
demo-tidb-initializer-jfxwl       1/1     Running            0          102m
demo-tikv-0                       1/2     CrashLoopBackOff   21         58m
demo-tikv-1                       1/2     CrashLoopBackOff   16         58m
demo-tikv-2                       1/2     CrashLoopBackOff   16         57m


kubectl describe pods demo-tikv-0 -n tidb
Name:               demo-tikv-0
Namespace:          tidb
Priority:           0
PriorityClassName:  <none>
Node:               umstor14/192.168.180.138
Start Time:         Wed, 06 Mar 2019 17:52:08 +0800
Labels:             app.kubernetes.io/component=tikv
                    app.kubernetes.io/instance=tidb-cluster
                    app.kubernetes.io/managed-by=tidb-operator
                    app.kubernetes.io/name=tidb-cluster
                    controller-revision-hash=demo-tikv-bffdb79d9
                    statefulset.kubernetes.io/pod-name=demo-tikv-0
Annotations:        pingcap.com/last-applied-configuration:
                      {"volumes":[{"name":"annotations","downwardAPI":{"items":[{"path":"annotations","fieldRef":{"fieldPath":"metadata.annotations"}}]}},{"name...
                    prometheus.io/path: /metrics
                    prometheus.io/port: 9091
                    prometheus.io/scrape: true
Status:             Running
IP:                 10.200.1.17
Controlled By:      StatefulSet/demo-tikv
Containers:
  tikv:
    Container ID:  docker://5b14f6d21e524992ea9974e2a1d61d40476cd328d77606d2b5a8afc7014fa563
    Image:         pingcap/tikv:v2.1.0
    Image ID:      docker-pullable://pingcap/tikv@sha256:7611c99f244fe537b7a00288b178187200611661503ca8401fe27f019e365db0
    Port:          20160/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      /usr/local/bin/tikv_start_script.sh
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 06 Mar 2019 17:52:26 +0800
      Finished:     Wed, 06 Mar 2019 17:52:26 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 06 Mar 2019 17:52:10 +0800
      Finished:     Wed, 06 Mar 2019 17:52:10 +0800
    Ready:          False
    Restart Count:  2
    Environment:
      NAMESPACE:              tidb (v1:metadata.namespace)
      CLUSTER_NAME:           demo
      HEADLESS_SERVICE_NAME:  demo-tikv-peer
      CAPACITY:               0
      TZ:                     UTC
    Mounts:
      /etc/podinfo from annotations (ro)
      /etc/tikv from config (ro)
      /usr/local/bin from startup-script (ro)
      /var/lib/tikv from tikv (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f4sff (ro)
  pushgateway:
    Container ID:   docker://8bc0e2e8f26588818bc107f9371ed08339907141eeb1338bfade613c4030b4d6
    Image:          prom/pushgateway:v0.3.1
    Image ID:       docker-pullable://prom/pushgateway@sha256:a108d9749fc0b9e6dac38c3c1dd612b24ff34f278078b0b70aba39c0aaced81e
    Port:           9091/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 06 Mar 2019 17:52:09 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     50m
      memory:  50Mi
    Environment:
      TZ:  UTC
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f4sff (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  tikv:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  tikv-demo-tikv-0
    ReadOnly:   false
  annotations:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      demo-tikv
    Optional:  false
  startup-script:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      demo-tikv
    Optional:  false
  default-token-f4sff:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-f4sff
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  20s               tidb-scheduler     Successfully assigned tidb/demo-tikv-0 to umstor14
  Normal   Pulled     19s               kubelet, umstor14  Container image "prom/pushgateway:v0.3.1" already present on machine
  Normal   Created    19s               kubelet, umstor14  Created container
  Normal   Started    19s               kubelet, umstor14  Started container
  Normal   Pulled     2s (x3 over 19s)  kubelet, umstor14  Container image "pingcap/tikv:v2.1.0" already present on machine
  Normal   Created    2s (x3 over 19s)  kubelet, umstor14  Created container
  Normal   Started    2s (x3 over 19s)  kubelet, umstor14  Started container
  Warning  BackOff    1s (x3 over 17s)  kubelet, umstor14  Back-off restarting failed container



kubectl get pv -n tidb
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                   STORAGECLASS    REASON   AGE
local-pv-1157f630   99Gi       RWO            Retain           Bound       tidb/pd-demo-pd-2       local-storage            84m
local-pv-1552be44   99Gi       RWO            Delete           Available                           local-storage            84m
local-pv-1ad1b13f   99Gi       RWO            Delete           Available                           local-storage            82m
local-pv-226a8e57   99Gi       RWO            Delete           Available                           local-storage            84m
local-pv-2ac2c85b   99Gi       RWO            Retain           Bound       tidb/pd-demo-pd-0       local-storage            84m
local-pv-4b3596b6   99Gi       RWO            Retain           Bound       tidb/tikv-demo-tikv-0   local-storage            84m
local-pv-5e37e14e   99Gi       RWO            Retain           Bound       tidb/tikv-demo-tikv-1   local-storage            82m
local-pv-6ee370e9   149Gi      RWO            Delete           Available                           local-storage            84m
local-pv-7437a961   99Gi       RWO            Retain           Bound       tidb/pd-demo-pd-1       local-storage            83m
local-pv-76003c42   99Gi       RWO            Delete           Available                           local-storage            82m
local-pv-79bdc895   99Gi       RWO            Retain           Bound       tidb/tikv-demo-tikv-2   local-storage            82m
local-pv-8a1a18e2   149Gi      RWO            Delete           Available                           local-storage            84m
local-pv-9c46be9d   99Gi       RWO            Delete           Available                           local-storage            84m
local-pv-c5fc6ec3   149Gi      RWO            Delete           Available                           local-storage            82m
local-pv-dcb06550   149Gi      RWO            Delete           Available                           local-storage            82m
local-pv-e9dfc52c   99Gi       RWO            Delete           Available                           local-storage            82m


kubectl get pvc -n tidb
NAME               STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pd-demo-pd-0       Bound    local-pv-2ac2c85b   99Gi       RWO            local-storage   105m
pd-demo-pd-1       Bound    local-pv-7437a961   99Gi       RWO            local-storage   105m
pd-demo-pd-2       Bound    local-pv-1157f630   99Gi       RWO            local-storage   105m
tikv-demo-tikv-0   Bound    local-pv-4b3596b6   99Gi       RWO            local-storage   82m
tikv-demo-tikv-1   Bound    local-pv-5e37e14e   99Gi       RWO            local-storage   82m
tikv-demo-tikv-2   Bound    local-pv-79bdc895   99Gi       RWO            local-storage   82m

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 25 (13 by maintainers)

Commits related to this issue

update toc (#299) — committed to pingcap/tidb-operator by DanielZhangQD 4 years ago

Most upvoted comments

Set LimitNOFILE and LimitNPROC to 1048576 in the file: /usr/lib/systemd/system/docker.service

weekface on Mar 6, 2019

I think we should update the corresponding section of setup.md, that seems out-dated:

Because TiDB by default will use at most 40960 file descriptors, the worker node and its Docker daemon’s ulimit must be configured to greater than 40960:

$ sudo vim /etc/systemd/system/docker.service

Set LimitNOFILE to equal or greater than 40960.

aylei on Mar 6, 2019