kube-httpcache: Unable to start new statefulset due to readiness probe

Describe the bug When setting a Kubernetes readiness probe, Varnish will never start when creating a new stateful set or the pod count goes to 0. The first pod will wait for the front end service to get a ready endpoint, but the ready endpoint depends on the server being active.

The readiness probe will work properly if you disable it on the first deploy, then enable it after that. The stateful set will however get into a bad state if all pods are restarted.

My settings could be wrong, but my understanding is that frontend-service should be set to the service in front of the Varnish cache?

W0824 17:09:56.720436       1 endpoints_watch.go:50] service 'api-varnish' has no endpoints

To Reproduce Steps to reproduce the behavior:

Create a new statefulset with a liveness and readiness probe:

    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 9102
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 9102
        scheme: HTTP
      initialDelaySeconds: 5
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5

The first pod will never become ready because it is watching its own frontend service and waiting for it to be active.

Expected behavior On first deploy or when pod count goes to 0, the first pod should become ready by either ignoring the front end service check or skipping it. This would allow subsequent pods to start up

Environment:

Kubernetes version: 1.16
kube-httpcache version: v0.3.2 (Though this issue existed pre 0.3.2 if you set the readiness probe to port 80)

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (4 by maintainers)

Most upvoted comments

Hello,

I created a solution with this resources:

Created a new headless service to statefulset with “publishNotReadyAddresses: true”;
Changed the " -frontend-service" to use the headless service
And added the livenessPRobe/readinessProbe using tcpSocket

With this “approach” the kube-httpcache will start correctly because they will find the headless frontend endpoint, but the original endpoint will receive requests from kubernetes ingress only when the health checks passed.

I hope this can help other people.

Regards

caiohasouza on Oct 21, 2020

@benjick so here is how i did it:

created a new headless service

apiVersion: v1
kind: Service
metadata:
  name: httpcache-headless
  namespace: namespace
  labels:
    app: cache
spec:
  publishNotReadyAddresses: true
  clusterIP: None
  selector:
    app: cache
  ports:
  - name: "http-varnish"
    port: 80
    targetPort: 80
  - name: "http-signaller"
    port: 8090
    targetPort: 8090

Then in the args you pass to the cache image, you point the frontend-service to the headless service

#...
containers:
        - name: cache
          image: quay.io/mittwald/kube-httpcache:stable
          imagePullPolicy: Always
          args:
          - -admin-addr=0.0.0.0
          - -admin-port=6083
          - -signaller-enable
          - -signaller-port=8090
          - -frontend-watch
          - -frontend-namespace=$(NAMESPACE)
          - -frontend-service=httpcache-headless
          - -frontend-portname=http-varnish
#...

I didn’t have to use tcpSocket for the readiness/liveness probe

        readinessProbe:
            httpGet:
              path: /varnishstatus
              port: 80
              scheme: HTTP
            initialDelaySeconds: 5
            failureThreshold: 5
            successThreshold: 1
            periodSeconds: 3
          livenessProbe:
            httpGet:
              path: /varnishstatus
              port: 80
              scheme: HTTP
            initialDelaySeconds: 30
            failureThreshold: 5
            successThreshold: 1
            periodSeconds: 5

the path /varnishstatus was a synthetic route that I had added to the vcl template

olamiwhat on Dec 8, 2020

@olamiwhat thank you 🙏 now it makes sense! I couldn’t figure out how to connect the two services, I didn’t realize I shouldn’t! Cheers!

benjick on Dec 8, 2020