serving: Stuck in RevisionMissing and "Unknown" state

In what area(s)?

/area networking

What version of Knative?

v0.11.0

Expected Behavior

Service is created successfully

Actual Behavior

Resources stuck in various bad states:

$ k get all
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/knative-service-hl52c-deployment-76bd8bfbdb-vphtq   2/2     Running   0          6m7s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/knative-service-hl52c-deployment   1/1     1            1           6m7s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/knative-service-hl52c-deployment-76bd8bfbdb   1         1         1       6m7s

NAME                                          URL                                                 LATESTCREATED           LATESTREADY   READY     REASON
service.serving.knative.dev/knative-service   http://knative-service.knative-1-8760.example.com   knative-service-hl52c                 Unknown   RevisionMissing

NAME                                                 CONFIG NAME       K8S SERVICE NAME   GENERATION   READY     REASON
revision.serving.knative.dev/knative-service-hl52c   knative-service                      1            Unknown

NAME                                                LATESTCREATED           LATESTREADY   READY     REASON
configuration.serving.knative.dev/knative-service   knative-service-hl52c                 Unknown

NAME                                        URL                                                 READY     REASON
route.serving.knative.dev/knative-service   http://knative-service.knative-1-8760.example.com   Unknown   RevisionMissing

apiVersion: v1
items:
- apiVersion: serving.knative.dev/v1
  kind: Revision
  metadata:
    creationTimestamp: "2019-12-19T23:19:27Z"
    generateName: knative-service-
    generation: 1
    labels:
      serving.knative.dev/configuration: knative-service
      serving.knative.dev/configurationGeneration: "1"
      serving.knative.dev/service: knative-service
    name: knative-service-hl52c
    namespace: knative-1-8760
    ownerReferences:
    - apiVersion: serving.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Configuration
      name: knative-service
      uid: 3f1f7137-89b6-45bc-88b0-babccd4d607a
    resourceVersion: "1065"
    selfLink: /apis/serving.knative.dev/v1/namespaces/knative-1-8760/revisions/knative-service-hl52c
    uid: 4677f591-05b9-447b-8b29-4abb8d4ff663
  spec:
    containerConcurrency: 0
    containers:
    - image: gcr.io/istio-testing/app:latest
      name: user-container
      readinessProbe:
        successThreshold: 1
        tcpSocket:
          port: 0
      resources: {}
    timeoutSeconds: 300
  status:
    conditions:
    - lastTransitionTime: "2019-12-19T23:19:28Z"
      reason: Deploying
      severity: Info
      status: Unknown
      type: Active
    - lastTransitionTime: "2019-12-19T23:19:28Z"
      status: Unknown
      type: ContainerHealthy
    - lastTransitionTime: "2019-12-19T23:19:28Z"
      status: Unknown
      type: Ready
    - lastTransitionTime: "2019-12-19T23:19:28Z"
      status: "True"
      type: ResourcesAvailable
    imageDigest: gcr.io/istio-testing/app@sha256:80790eb8ab6a4453e3fa8c2bc84e2d6b0cef095c4ceec79bc17cbf01fceb72cc
    logUrl: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'4677f591-05b9-447b-8b29-4abb8d4ff663',type:phrase))))
    observedGeneration: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

apiVersion: v1
items:
- apiVersion: serving.knative.dev/v1
  kind: Configuration
  metadata:
    annotations:
      serving.knative.dev/forceUpgrade: "true"
    creationTimestamp: "2019-12-19T23:19:27Z"
    generation: 1
    labels:
      serving.knative.dev/route: knative-service
      serving.knative.dev/service: knative-service
    name: knative-service
    namespace: knative-1-8760
    ownerReferences:
    - apiVersion: serving.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: knative-service
      uid: 0d7555d6-d266-4886-884a-440e789f1a49
    resourceVersion: "1042"
    selfLink: /apis/serving.knative.dev/v1/namespaces/knative-1-8760/configurations/knative-service
    uid: 3f1f7137-89b6-45bc-88b0-babccd4d607a
  spec:
    template:
      metadata:
        creationTimestamp: null
      spec:
        containerConcurrency: 0
        containers:
        - image: gcr.io/istio-testing/app:latest
          name: user-container
          readinessProbe:
            successThreshold: 1
            tcpSocket:
              port: 0
          resources: {}
        timeoutSeconds: 300
  status:
    conditions:
    - lastTransitionTime: "2019-12-19T23:19:27Z"
      status: Unknown
      type: Ready
    latestCreatedRevisionName: knative-service-hl52c
    observedGeneration: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Basically everything is unknown with no real indication about what is going wrong

Steps to Reproduce the Problem

I am trying to get some knative smoke tests integrated into Istio’s tests so we don’t break things accidentally. See PR https://github.com/istio/istio/pull/19675. It seems fairly reproducible on a fresh cluster, running those steps.

I am probably doing something wrong, but all of the status messages and logs are not leading me in the right direction

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 3
  • Comments: 43 (18 by maintainers)

Most upvoted comments

Same problem on v0.12.0,kubeadm(v.17.0),centos7.7.1908. Pods info below

[root@kn12 ~]# kubectl get po --all-namespaces
NAMESPACE         NAME                                                      READY   STATUS    RESTARTS   AGE
default           stock-service-example-first-deployment-5d95c6bc8b-tnzww   2/2     Running   0          14m
istio-system      istio-ingressgateway-5c4758f88f-sdh7h                     1/1     Running   0          20m
istio-system      istio-pilot-55d5864f44-pqtpn                              1/1     Running   0          20m
knative-serving   activator-6654488fb4-vrrrb                                1/1     Running   0          50m
knative-serving   autoscaler-7c78fb45b5-hhcmh                               1/1     Running   0          50m
knative-serving   autoscaler-hpa-84dcb8c78-tsrr5                            1/1     Running   0          50m
knative-serving   controller-557bf98b95-46zsx                               1/1     Running   0          50m
knative-serving   networking-istio-7f57947886-v6xm7                         1/1     Running   0          50m
knative-serving   webhook-bb564cf84-7p2wq                                   1/1     Running   0          50m
kube-system       coredns-6955765f44-fhz9v                                  1/1     Running   0          17h
kube-system       coredns-6955765f44-gn6d2                                  1/1     Running   0          17h
kube-system       etcd-kn12                                                 1/1     Running   1          17h
kube-system       kube-apiserver-kn12                                       1/1     Running   1          17h
kube-system       kube-controller-manager-kn12                              1/1     Running   1          17h
kube-system       kube-flannel-ds-amd64-nlfws                               1/1     Running   0          16m
kube-system       kube-proxy-kgk2h                                          1/1     Running   1          17h
kube-system       kube-scheduler-kn12                                       1/1     Running   1          17h
[root@kn12 ~]#

sample app info below

[root@kn12 ~]# kubectl get all
NAME                                                          READY   STATUS    RESTARTS   AGE
pod/stock-service-example-first-deployment-5d95c6bc8b-tnzww   2/2     Running   0          10m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   17h

NAME                                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/stock-service-example-first-deployment   1/1     1            1           10m

NAME                                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/stock-service-example-first-deployment-5d95c6bc8b   1         1         1       10m

NAME                                                       CONFIG NAME             K8S SERVICE NAME   GENERATION   READY     REASON
revision.serving.knative.dev/stock-service-example-first   stock-service-example                      1            Unknown   Deploying

NAME                                                      LATESTCREATED                 LATESTREADY   READY     REASON
configuration.serving.knative.dev/stock-service-example   stock-service-example-first                 Unknown

NAME                                              URL                                                READY     REASON
route.serving.knative.dev/stock-service-example   http://stock-service-example.default.example.com   Unknown   RevisionMissing

NAME                                                URL                                                LATESTCREATED                 LATESTREADY   READY     REASON
service.serving.knative.dev/stock-service-example   http://stock-service-example.default.example.com   stock-service-example-first                 Unknown   RevisionMissing
[root@kn12 ~]#

Revision describe info

[root@kn12 ~]# kubectl describe revision stock-service-example-first
Name:         stock-service-example-first
Namespace:    default
Labels:       serving.knative.dev/configuration=stock-service-example
              serving.knative.dev/configurationGeneration=1
              serving.knative.dev/service=stock-service-example
Annotations:  <none>
API Version:  serving.knative.dev/v1
Kind:         Revision
Metadata:
  Creation Timestamp:  2020-02-25T02:40:57Z
  Generation:          1
  Owner References:
    API Version:           serving.knative.dev/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Configuration
    Name:                  stock-service-example
    UID:                   78996df2-33c2-469d-a0f9-772656da4e16
  Resource Version:        24611
  Self Link:               /apis/serving.knative.dev/v1/namespaces/default/revisions/stock-service-example-first
  UID:                     b1414971-19de-43dd-87ed-e50f59bff476
Spec:
  Container Concurrency:  0
  Containers:
    Env:
      Name:   RESOURCE
      Value:  stock
    Image:    registry.cn-chengdu.aliyuncs.com/kn-sample/rest-api-go
    Name:     user-container
    Readiness Probe:
      Http Get:
        Path:             /
        Port:             0
      Period Seconds:     3
      Success Threshold:  1
    Resources:
  Timeout Seconds:  300
Status:
  Conditions:
    Last Transition Time:  2020-02-25T02:40:58Z
    Reason:                Deploying
    Severity:              Info
    Status:                Unknown
    Type:                  Active
    Last Transition Time:  2020-02-25T02:40:58Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  ContainerHealthy
    Last Transition Time:  2020-02-25T02:40:58Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  Ready
    Last Transition Time:  2020-02-25T02:40:58Z
    Reason:                Deploying
    Status:                Unknown
    Type:                  ResourcesAvailable
  Image Digest:            registry.cn-chengdu.aliyuncs.com/kn-sample/rest-api-go@sha256:a12f6f2a6151f19717ec7ba1a91dc1cd2d40acbeed9fd344d3a2a511dc29565e
  Log URL:                 http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'b1414971-19de-43dd-87ed-e50f59bff476',type:phrase))))
  Observed Generation:     1
Events:
  Type     Reason         Age   From                 Message
  ----     ------         ----  ----                 -------
  Warning  InternalError  12m   revision-controller  failed to update deployment "stock-service-example-first-deployment": Operation cannot be fulfilled on deployments.apps "stock-service-example-first-deployment": the object has been modified; please apply your changes to the latest version and try again
+1
ysjjovo on Feb 25, 2020

Same problem on v0.11.0,k8s v.15.4(binary setup),centos7.7.1908. but kubeadm(v1.17.0) works fine.

+1
ysjjovo on Feb 23, 2020

Same problem on v0.12.0 - totally a showstopper

+1
JCzz on Feb 22, 2020
Read more comments on GitHub
← serving: SinkBinding doesn't work with KServices that uses BYO revision names
serving: Tag to digest resolving is not working in Knative 1.3.0 →