origin: GlusterFS volume error : glusterfs: failed to get endpoints

I followed the instructions from [1] and [2] to setup Gluster FS PV but when I try to use the PVC in a pod it fails.

The error in node logs is :

Dec 15 21:48:52 node-compute-8c258955.dev.aws.qtz.io origin-node[405]: E1215 21:48:52.824595     405 glusterfs.go:89] glusterfs: failed to get endpoints glusterfs-dev[endpoints "glusterfs-dev" not found]
Dec 15 21:48:52 node-compute-8c258955.dev.aws.qtz.io origin-node[405]: E1215 21:48:52.832670     405 glusterfs.go:89] glusterfs: failed

But the “glusterfs-dev” service and endpoint exists in the “default” namespace:

$ oc get service glusterfs-dev -n default -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2015-12-15T21:32:23Z
  name: glusterfs-dev
  namespace: default
  resourceVersion: "2894020"
  selfLink: /api/v1/namespaces/default/services/glusterfs-dev
  uid: 5424624c-a373-11e5-b562-06302f7b1fe5
spec:
  clusterIP: 172.30.224.96
  portalIP: 172.30.224.96
  ports:
  - port: 1
    protocol: TCP
    targetPort: 1
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

$ oc get endpoints glusterfs-dev -n default -o yaml
apiVersion: v1
kind: Endpoints
metadata:
  creationTimestamp: 2015-12-15T11:03:46Z
  name: glusterfs-dev
  namespace: default
  resourceVersion: "2839215"
  selfLink: /api/v1/namespaces/default/endpoints/glusterfs-dev
  uid: 82f60af3-a31b-11e5-b562-06302f7b1fe5
subsets:
- addresses:
  - ip: 172.18.1.89
  - ip: 172.18.1.90
  ports:
  - port: 1
    protocol: TCP

It looks like the glusterfs plugin looks for the endpoint in the pod namespace [3] rather that the “default” namespace as documented.

I tried creating the endpoint in the project and it worked.

Is this a document bug or a regression ? Creating endpoints per project is cumbersome as the endpoints need glusterfs cluster IPs and do not work with hostname (as per [1]) and any change in glusterfs cluster will mean that every project on the cluster needs to be updated.

[1] https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_glusterfs.html [2] https://github.com/kubernetes/kubernetes/issues/12964 [3] https://github.com/openshift/origin/blob/master/Godeps/_workspace/src/k8s.io/kubernetes/pkg/volume/glusterfs/glusterfs.go#L86

About this issue

Original URL
State: closed
Created 9 years ago
Reactions: 2
Comments: 47 (29 by maintainers)

Commits related to this issue

Introduce endpointnamespace in glustervolumesource Fix for : https://github.com/openshift/origin/issues/6331 Signed-off-by: Humble Chirammal <hchiramm@redhat.com> — committed to humblec/kubernetes by humblec 8 years ago
Introduce endpointnamespace in glustervolumesource Fix for : https://github.com/openshift/origin/issues/6331 — committed to humblec/kubernetes by humblec 8 years ago
Introduce endpointnamespace in glustervolumesource Fix for : https://github.com/openshift/origin/issues/6331 Signed-off-by: Humble Chirammal <hchiramm@redhat.com> — committed to humblec/kubernetes by humblec 8 years ago

Most upvoted comments

Hello,

A customer using glusterfs in a pretty large OpenShift cluster (3k containers, > 400 PVs) here 😃

No I think this is not closed by kubernetes/kubernetes#31854. With heketi you just create/delete the service & endpoints dynamically, but the problem still remains.

I still don’t like that gluster pv definition is different from all the other storage types. Why are not all the information inside the gluster pv object like with other storage types? The current solution has several downsides:

Mixes things that cluster admins (PV) manage with things project admins manage (Services, Endpoints). Also exposes information to the projects that are not relevant to them.
Noise in UI/exports/templates/cli and so on
Redundancy (we have >500 namespaces) so we would have 500x the same services/endpoints
Updating gluster server IPs is a pain! (even if heketi takes care of parts of it)
Project admins will (an already did) delete the configs by mistake (read-only seems like a hack…)

I know about your limitations with changing the existing object. Here a few ideas:

kind: PersistentVolume
spec:
  ...
  glusterfs:
    path: MyVolume
    endpoints: glusterfs-cluster
    namespace: global-gluster-config  # new optional field

If this field is not specified it takes the data from the current namespace. If specified and no access right just write an event with an error message. We would have no problem to add this config to one of our global namespaces where everyone has read permissions.

Or even better, add the information directly to the pv object:

kind: PersistentVolume
spec:
  ...
  glusterfs:
    path: MyVolume
    glusterIPs:  # new optional list
    - 10.1.1.1
    - 10.1.1.2
    endpoints: # make optional if possible or we just put a dummy string in there

Then if glusterIPs is present, take these IPs, otherwise fallback to current solution.

DNS seems also like a good way to go. I would like to “re-activate” the discussion around this topic.

ReToCode on Sep 8, 2017

HI @ReToCode, you got most of it, one blocker was backward compatibility or the difficulty to just discard endpoint from spec. What we are working on atm is endpoint namespace in PV spec and we are on it, it should take are of this issue.

humblec on Sep 8, 2017

yes, service and endpoints live in the same namespace.

rootfs on Apr 7, 2016