vault-helm: Consul Client Not Running on HA Mode of Vault

Vault should have consul client running, But, consul is not running,

kubectl exec vault-helm-0 -it sh
/ $ ps -ef | grep consul
28322 vault     0:00 grep consul

I have deployed Vault on k8s in HA and want to use consul as storage. The pods are not running.

vault-helm-0                           0/1     Running   0          162m
vault-helm-1                           0/1     Running   0          162m
vault-helm-2                           0/1     Running   0          162m.```

Logs

WARNING! Unable to read storage migration status.
2020-04-23T18:27:15.699Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-04-23T18:27:15.701Z [WARN]  storage migration check error: error="Get http://127.0.0.1:8500/v1/kv/vault/core/migration: dial tcp 127.0.0.1:8500: connect: connection refused"

Describing the Pod

Name:               vault-helm-0
Namespace:          spr-xxx
Priority:           0
PriorityClassName:  <none>
Node:               qa4-apps-k8s-node-202003241110-10-1a/10.xx.xxx.xxx
Start Time:         Thu, 23 Apr 2020 18:27:13 +0000
Labels:             app.kubernetes.io/instance=vault-helm
                    app.kubernetes.io/name=vault
                    component=server
                    controller-revision-hash=vault-helm-764cc498f5
                    helm.sh/chart=vault-0.5.0
                    statefulset.kubernetes.io/pod-name=vault-helm-0
Annotations:        cni.projectcalico.org/podIP: 192.168.43.48/32
                    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu, memory request for container vault; cpu, memory limit for container vault
Status:             Running
IP:                 192.168.xx.xx
Controlled By:      StatefulSet/vault-helm
Containers:
  vault:
    Container ID:  docker://a0e8c5b0ac6c181ea0b4a8871edf4a41967780520e3ff2be1c3d7b183518fe60
    Image:         vault:1.3.2
    Image ID:      docker-pullable://vault@sha256:cf9d54f9a5ead66076066e208dbdca2094531036d4b053c596341cefb17ebf95
    Ports:         8200/TCP, 8201/TCP, 8202/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
    Args:
      sed -E "s/HOST_IP/${HOST_IP?}/g" /vault/config/extraconfig-from-values.hcl > /tmp/storageconfig.hcl;
      sed -Ei "s/POD_IP/${POD_IP?}/g" /tmp/storageconfig.hcl;
      /usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

    State:          Running
      Started:      Thu, 23 Apr 2020 18:27:15 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      500m
      memory:   256Mi
    Readiness:  exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=5s period=3s #success=1 #failure=2
    Environment:
      HOST_IP:               (v1:status.hostIP)
      POD_IP:                (v1:status.podIP)
      VAULT_K8S_POD_NAME:   vault-helm-0 (v1:metadata.name)
      VAULT_K8S_NAMESPACE:  spr-xxx (v1:metadata.namespace)
      VAULT_ADDR:           https://127.0.0.1:8200
      VAULT_API_ADDR:       https://$(POD_IP):8200
      SKIP_CHOWN:           true
      SKIP_SETCAP:          true
      HOSTNAME:             vault-helm-0 (v1:metadata.name)
      VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-helm-internal:8201
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from vault-helm-token-ptt4p (ro)
      /vault/config from config (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      vault-helm-config
    Optional:  false
  vault-helm-token-ptt4p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  vault-helm-token-ptt4p
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From                                           Message
  ----     ------     ----                ----                                           -------
  Normal   Scheduled  89s                 default-scheduler                              Successfully assigned spr-ops/vault-helm-0 to qa4-apps-k8s-node-202003241110-10-1a
  Normal   Pulled     88s                 kubelet, k8s-node-202003241110-10-1a  Container image "vault:1.3.2" already present on machine
  Normal   Created    87s                 kubelet, k8s-node-202003241110-10-1a  Created container
  Normal   Started    87s                 kubelet, k8s-node-202003241110-10-1a  Started container
  Warning  Unhealthy  18s (x22 over 81s)  kubelet, k8s-node-202003241110-10-1a  Readiness probe failed: Error checking seal status: Get https://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: connect: connection refused

Also, I am receiving error even while passing the setting the variable env why?

kubectl exec vault-helm-0 -it sh
/ $ export VAULT_ADDR=http://127.0.0.1:8200
/ $ vault -v
Vault v1.3.2
/ $ vault operator init -n 1 -t 1
Error initializing: Put http://127.0.0.1:8200/v1/sys/init: dial tcp 127.0.0.1:8200: connect: connection refused

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 3
  • Comments: 16 (4 by maintainers)

Most upvoted comments

I got the same issue . $ oc exec -it vault-0 vault status Error checking seal status: Get “http://127.0.0.1:8200/v1/sys/seal-status”: dial tcp 127.0.0.1:8200: connect: connection refused command terminated with exit code 1 when I look into pod logs I can see below 2020-11-04T12:23:17.945Z [WARN] storage migration check error: error=“Get “http://10.160.225.18:8500/v1/kv/vault/core/migration”: dial tcp 10.160.225.18:8500: connect: connection refused” So what I understood is , 10.160.225.18(HOST_IP) is my worker node where consul server pod is running , vault is not connecting to consul server with HOST_IP with 8500 port number , and below is my values .yaml storage “consul” { path = “vault/” address = “HOST_IP:8500”

what the work around I did was , I Changed HOST_IP:8500 to my consul SERVICE_IP , hence it is headless service , service ip not generated so I have given my consul service name , in my case my consul service name is “consul-server” balaji@DESKTOP-O8C6N39:~/vault$ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE consul-dns ClusterIP 172.21.33.6 53/TCP,53/UDP 2d consul-server ClusterIP None 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP

values.yaml

storage “consul” { path = “vault” address = “consul-server:8500”

Then vault was deployed and working fine SO finally my vault HA with backend consul storage is working perfectly .

Hi @rdeb22, can you make the following changes to your config?

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }