Same with rancher / rancher: v2.2.3 when scaling down / up “rabbitmq-ha” (from helm) as described by @bentastic27

The issue is that on upscaling the service, the “RABBITMQ_NODENAME” environnent variable seems to be evaluated before “MY_POD_NAME” and therefore RABBITMQ_NODENAME could not rely on MY_POD_NAME which is undefined.

Find below is the interesting part of my StatefulSet to undestand the issue :

Erroneous config file modified by rancher :

apiVersion: apps/v1
kind: StatefulSet
spec:
  template:
    spec:
      containers:
      - env:
        - name: K8S_HOSTNAME_SUFFIX
          value: .rabbitmq-ha-discovery.production.svc.cluster.local
        - name: K8S_SERVICE_NAME
          value: rabbitmq-ha-discovery
        - name: RABBITMQ_NODENAME
          value: rabbit@$(MY_POD_NAME).rabbitmq-ha-discovery.production.svc.cluster.local
        - name: RABBITMQ_USE_LONGNAME
          value: 'true'
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name

RABBITMQ_NODENAME depends on MY_POD_NAME, it must be defined after.

Modified configuration file that works :

apiVersion: apps/v1
kind: StatefulSet
spec:
  template:
    spec:
      containers:
      - env:
        - name: K8S_HOSTNAME_SUFFIX
          value: .rabbitmq-ha-discovery.production.svc.cluster.local
        - name: K8S_SERVICE_NAME
          value: rabbitmq-ha-discovery
        - name: RABBITMQ_USE_LONGNAME
          value: 'true'
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: RABBITMQ_NODENAME
          value: rabbit@$(MY_POD_NAME).rabbitmq-ha-discovery.production.svc.cluster.local

This behavior is not limited just to rabbit-mq or apps deployed via helm charts. We’re facing the same when working with StatefulSet, Deployment etc. Issue observed on Rancher 4.2.5.

I believe it is related to how Rancher handles yaml specification rendering before deploying it to Kubernetes. From what I can see by testing is that the variables set in Environment variables section will be placed above those set in Inject values from another resource section in the final specification. To illustrate here’s example screenshot:

How to reproduce And here is resulting yaml file of that same deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    field.cattle.io/creatorId: user-4tvsz
  creationTimestamp: "2020-07-27T07:28:23Z"
  generation: 1
  labels:
    cattle.io/creator: norman
    workload.user.cattle.io/workloadselector: deployment-p12-dev-test
  name: test
  namespace: p12-dev
  resourceVersion: "69551669"
  selfLink: /apis/apps/v1/namespaces/p12-dev/deployments/test
  uid: b3849b19-a95a-4476-a028-55589146a5ca
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-p12-dev-test
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cattle.io/timestamp: "2020-07-27T07:28:22Z"
      creationTimestamp: null
      labels:
        workload.user.cattle.io/workloadselector: deployment-p12-dev-test
    spec:
      containers:
      - env:
        - name: TEST_VAR
          value: $(DEPEND_ON_REF_NS)
        - name: DEPEND_ON_REF_NS
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: nginx
        imagePullPolicy: Always
        name: test
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities: {}
          privileged: false
          readOnlyRootFilesystem: false
          runAsNonRoot: false
        stdin: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        tty: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-07-27T07:28:33Z"
    lastUpdateTime: "2020-07-27T07:28:33Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-07-27T07:28:23Z"
    lastUpdateTime: "2020-07-27T07:28:33Z"
    message: ReplicaSet "test-855fbf544f" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

And here is resulting environment inside the running container, not exactly what we wanted right?

# env | grep DEP
TEST_VAR=$(DEPEND_ON_REF_NS)
DEPEND_ON_REF_NS=p12-dev

Workaround for our case Currently we can simply deploy via yaml using Kubectl. Here’s a simple deployment spec, resulting in same deployment as above via Rancher UI. Note that order variables are defined here is reversed from what Rancher has produced:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test
  namespace: p12-dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - env:
        - name: DEPEND_ON_REF_NS
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: TEST_VAR
          value: $(DEPEND_ON_REF_NS)
        image: nginx
        imagePullPolicy: Always
        name: test

This time result in running container is what we expect it to be:

# env | grep "TEST\|DEP"
TEST_VAR=p12-dev
DEPEND_ON_REF_NS=p12-dev

The problem with this approach is that whenever someone would edit the deployment via Rancher UI or use Redeploy functionality, the specification would be re-rendered in Rancher’s current style, resulting in broken deployment.

Expected behavior We would certainly expect that we are able to use dependent variables as described in Kubernetes documentation without fear of our deployments getting broken by someone editing them in Rancher UI or Redeploying. I think a very simple solution that should work is to change the order Rancher places variable declarations in the specifications by always placing explicitly defined variables after those injected from other sources (field refs, resource refs etc.).

I am experiencing the same issue, and it’s been particularly frustrating since no matter how many times we attempt to update the file to have the properties in the correct order it just immediately reverts to the existing file. What we end up with is pods that stay broken and we can’t scale back up because the first pod is borked, requiring us to completely remove and recreate the cluster in order to get it working again. In fairness we are running it as a multi-cluster app but I am not sure how that would make a difference. Any insight would be appreciated!

⚠️ Update: this has gotten so out of control. Now it seems that deleting any pod will prevent any new pods from ever being created. It seems the only way to “resolve” this issue is to completely nuke the application from the cluster (remove target from multi-cluster application) and then recreate it from scratch (add target back to multi-cluster application).

Here is the current scenario I just tried:

1. Have 5 pods running:
   • rabbitmq-ha-p-ndvdf-0 - Failed due to previously deleting the pod in hopes it would come back
   • rabbitmq-ha-p-ndvdf-1 - Healthy
   • rabbitmq-ha-p-ndvdf-2 - Healthy
   • rabbitmq-ha-p-ndvdf-3 - Healthy
   • rabbitmq-ha-p-ndvdf-4 - Healthy
2. Wondering if perhaps it was just pod 0 for some reason, I deleted rabbitmq-ha-p-ndvdf-4
3. The cluster doesn’t even attempt to recreate pod 4 again since pod 0 is in a crash/reboot loop due to the error this issue is about:

/opt/rabbitmq/sbin/rabbitmq-server: eval: line 1: MY_POD_NAME: not found

I guess I will attempt to remove the project from the multi-cluster app and then re-add it so it creates it from scratch. Once that’s done I’ll try deleting pod 3 to see if it’s all pods that are affected or just 0 for some reason.

⚠️ BIG UPDATE 2: OMG! This appears to be a zero-index issue!!! I did what I said I would do above, delete pod 4 instead of pod 0, and it worked on the first try in less than 30 seconds! Screenshots incoming:

Starting state. 5 healthy pods, up and running for 5 days

Preparing to delete pod 4

Pod 4 being removed

Pod 4 spinning back up to meet required 5 pods (this is where it always fails)

Except this time it didn’t! 🥳 🍾

⚠️ Update 3: Nope, failure. I took it one step further and removed pods 1-4 (still didn’t touch pod 0) and now all the pods I deleted are failing.

How is it even conceivably possible that this chart is in wide use?!? This is whatever the opposite of high availability is.

I am now forced to copy the chart into our own private repo and modify it until it hopefully works. I’ll probably break even more than is already broken, but clearly this chart won’t get fixed anytime soon.

I use rancher v2.5.8 and I have the same problem

Oh, this github issue. I keep hoping never to see it again and yet here we are. Totally surreal.

Is the rabbitmq-ha chart abandoned or something? How is it possible that this is still a thing?

@rmalchow i tested it on 2.4.5, there is still a problem