keda: Scaling from 0->N doesn't get initiated on time/at all

We have a scaledobject based on GCP PubSub. The PubSub subscription has reached the number of messages to have the deployment scaled up, but it doesn’t happen on time. Also, yesterday it didn’t happen at all on the same scaledobject, until we restarted the keda operator- It reached 27000 messages (Scaledobject is defined to get triggered on value of 50).

Expected Behavior

Once Pubsub subscription reaches the number of messages as defined in the scaledobject, the deployment needs to get scaled up from 0->N, based on the polling interval of course.

Actual Behavior

First issue- Doesn’t scale on time- Deployment got scaled to 0 on timestamp 1598771940.466385 (07:19:00.466 AM GMT). Pubsub messages show an increase over 50 messages (the value that’s defined in the scaledobject) on 07:21 AM GMT, however at 07:38 GMT it reached 972 messages and only then it got scaled up, a.k.a after 17 minutes of a high count of messages (relatively to the scaledobject configuration).

I would expect it to scale up back to N replicas after 7 minutes the latest - 2 min of polling interval + 5 minutes of the cooling period.

Second issue - Didn’t scale at all- Yesterday it didn’t scale back up at all (after almost 4 hours of high count of messages) until we restarted the Keda operator. Messages count on the subscription started increasing since 13:56 GMT.

Deployment got scaled to 0 on timestamp 1598709286.193081 (13:54:46.193 PM GMT). Pubsub messages show an increase over 50 messages (the value that’s defined in the scaledobject) on 13:56 PM GMT, however at 17:55 GMT it reached 27,063 messages and only after we restarted Keda operator on 17:53 PM GMT, it scaled the pods up to 1 replica.

I would expect it to scale up back to N replicas at 14:03 PM GMT the latest - 2 min of polling interval + 5 minutes of the cooling period. BUT it didn’t scale at all.

LOGS- First issue- Doesn’t scale on time- From Keda logs- scaling to 0 on 7:19 AM GMT- {"level":"info","ts":1598771940.466385,"logger":"scalehandler","msg":"Successfully scaled deployment to 0 replicas","ScaledObject.Namespace":"prod-us-log-forwarding","ScaledObject.Name":"lf-scaledobject-535176920","ScaledObject.ScaleType":"deployment","Deployment.Namespace":"prod-us-log-forwarding","Deployment.Name":"log-forwarding-535176920"}

Any other references to this scaledobject after that- {"level":"info","ts":1598771965.0714204,"logger":"controller_scaledobject","msg":"Reconciling ScaledObject","Request.Namespace":"prod-us-log-forwarding","Request.Name":"lf-scaledobject-535176920"} {"level":"info","ts":1598771965.071469,"logger":"controller_scaledobject","msg":"Detected ScaleType = Deployment","Request.Namespace":"prod-us-log-forwarding","Request.Name":"lf-scaledobject-535176920"}

we have more of those reconciling…detected… and then at 07:38:50.164 AM- {"level":"info","ts":1598773130.1646843,"logger":"scalehandler","msg":"Successfully updated deployment","ScaledObject.Namespace":"prod-us-log-forwarding","ScaledObject.Name":"lf-scaledobject-535176920","ScaledObject.ScaleType":"deployment","Deployment.Namespace":"prod-us-log-forwarding","Deployment.Name":"log-forwarding-535176920","Original Replicas Count":0,"New Replicas Count":1}

Nothing in between.

Second issue- Doesn’t scale at all- From Keda logs- As for yesterday in which it didn’t scale at all, only relevant errors I see in the logs are- 13:55:13 GMT- Worth mentioning this error is noticed on the logs also right before… a.k.a right before it got successfully scaled to 0. error: "rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": dial tcp: i/o timeout" level: "error" logger: "gcp_pub_sub_scaler" msg: "error getting Active Status" stacktrace: "github.com/go-logr/zapr.(*zapLogger).Error /Users/zroubali/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128 github.com/kedacore/keda/pkg/scalers.(*pubsubScaler).IsActive keda/pkg/scalers/gcp_pub_sub_scaler.go:88 github.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScaleDeployment keda/pkg/handler/scale_loop.go:119 github.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScale keda/pkg/handler/scale_loop.go:45 github.com/kedacore/keda/pkg/handler.(*ScaleHandler).HandleScaleLoop keda/pkg/handler/scale_loop.go:28" ts: 1598720729.596498 14:10:04 GMT - error: "rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: Post "https://oauth2.googleapis.com/token": unexpected EOF" level: "error" logger: "gcp_pub_sub_scaler" msg: "error getting Active Status" stacktrace: "github.com/go-logr/zapr.(*zapLogger).Error /Users/zroubali/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128 github.com/kedacore/keda/pkg/scalers.(*pubsubScaler).IsActive keda/pkg/scalers/gcp_pub_sub_scaler.go:88 github.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScaleDeployment keda/pkg/handler/scale_loop.go:119 github.com/kedacore/keda/pkg/handler.(*ScaleHandler).handleScale keda/pkg/handler/scale_loop.go:45 github.com/kedacore/keda/pkg/handler.(*ScaleHandler).HandleScaleLoop keda/pkg/handler/scale_loop.go:28" ts: 1598710204.4026008

These 2 errors types gets reported quite often.

Steps to Reproduce the Problem

I don’t know how this can be reproduced, but I’ll just share the configurations I have.

Scaledobject- (Don’t mind the wrong indentation… Happened when pasted here) apiVersion: keda.k8s.io/v1alpha1 kind: ScaledObject metadata: creationTimestamp: “2020-08-23T10:26:34Z” finalizers: finalizer.keda.k8s.io generation: 2 labels: deploymentName: log-forwarding-535176920 name: lf-scaledobject-535176920 namespace: prod-us-log-forwarding resourceVersion: “85343895” selfLink: /apis/keda.k8s.io/v1alpha1/namespaces/prod-us-log-forwarding/scaledobjects/lf-scaledobject-535176920 uid: 1e802b1a-e52b-11ea-8145-42010ab50004 spec: cooldownPeriod: 300 maxReplicaCount: 5 minReplicaCount: 0 pollingInterval: 120 scaleTargetRef: deploymentName: log-forwarding-535176920 triggers: metadata: credentials: GENERIC_GOOGLE_CREDENTIALS subscriptionName: lf-data-535176920-sub subscriptionSize: “50” type: gcp-pubsub

Specifications

• KEDA Version: 1.5.0.
• Platform & Version: GCP GKE, not sure which version is required here.
• Kubernetes Version: 1.14.10-gke.42.
• Scaler(s): Scaledobject.