kubernetes: Pods test failures after master upgrade 0.19.3 → 0.21.2

Conditions

  • e2e test version: 0.19.3
  • nodes version: 0.19.3
  • master version: 0.21.2

Tests

  • [FLAKE] Pods should be restarted with a docker exec “cat /tmp/health” liveness probe
  • [FLAKE] Pods should not be restarted with a docker exec “cat /tmp/health” liveness probe
  • [FLAKE] Pods should be restarted with a /healthz http liveness probe
  • [FAIL] Pods should be updated

Flake history

flakes ### Output #### Pods should be restarted with a docker exec "cat /tmp/health" liveness probe

Pod “liveness-exec” is forbidden: no API token found for service account e2e-test-5d05436c-2a5a-11e5-916e-42010af01555/default, retry after the token is automatically created and added to the service account

/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/
go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:451
creating pod liveness-exec
Expected error:
    <*errors.StatusError | 0xc208bcac80>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Pod \"liveness-exec\" is forbidden: no API token found for service account
e2e-test-5d05436c-2a5a-11e5-916e-42010af01555/default, retry after the token
is automatically created and added to the service account",
            Reason: "Forbidden",
            Details: {
                Name: "liveness-exec",
                Kind: "Pod",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 403,
        },
    }
    Pod "liveness-exec" is forbidden: no API token found for service account
e2e-test-5d05436c-2a5a-11e5-916e-42010af01555/default, retry after the token
is automatically created and added to the service account
not to have occurred

Pods should not be restarted with a docker exec “cat /tmp/health” liveness probe

Pod “liveness-exec” is forbidden: no API token found for service account e2e-test-10b92925-2b3f-11e5-8ace-42010af01555/default, retry after the token is automatically created and added to the service account

/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/
go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:477
creating pod liveness-exec
Expected error:
    <*errors.StatusError | 0xc20965a280>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Pod \"liveness-exec\" is forbidden: no API token found for service
account e2e-test-10b92925-2b3f-11e5-8ace-42010af01555/default, retry after the token
is automatically created and added to the service account",
            Reason: "Forbidden",
            Details: {
                Name: "liveness-exec",
                Kind: "Pod",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 403,
        },
    }
    Pod "liveness-exec" is forbidden: no API token found for service
account e2e-test-10b92925-2b3f-11e5-8ace-42010af01555/default, retry after the token
is automatically created and added to the service account
not to have occurred

Pods should be restarted with a /healthz http liveness probe

Pod “liveness-http” is forbidden: no API token found for service account e2e-test-50513bd8-2a76-11e5-948c-42010af01555/default, retry after the token is automatically created and added to the service account

/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:504
creating pod liveness-http
Expected error:
    <*errors.StatusError | 0xc2081ffe00>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Pod \"liveness-http\" is forbidden: no API token found for service account e2e-test-1969a2b3-2a04-11e5-9a2a-42010af01555/default, retry after the token is automatically created and added to the service account",
            Reason: "Forbidden",
            Details: {
                Name: "liveness-http",
                Kind: "Pod",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 403,
        },
    }
    Pod "liveness-http" is forbidden: no API token found for service account e2e-test-1969a2b3-2a04-11e5-9a2a-42010af01555/default, retry after the token is automatically created and added to the service account

Pods should be updated

may not update fields other than container.image

/go/src/github.com/GoogleCloudPlatform/kubernetes/output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:338 failed to update pod: Pod “pod-update-08769420-2b44-11e5-b3e2-42010af01555” is invalid: spec: invalid value '{Volumes:[{Name:default-token-7sezo VolumeSource:{HostPath:<nil> EmptyDir:<nil> GCEPersistentDisk:<nil> AWSElasticBlockStore:<nil> GitRepo:<nil> Secret:<>(0xc209b18b30){SecretName:default-token-7sezo} NFS:<nil> ISCSI:<nil> Glusterfs:<nil> PersistentVolumeClaim:<nil> RBD:<nil>}}] Containers:[{Name:nginx Image:gcr.io/google_containers/nginx:1.7.9 Command:<nil> Args:<nil> WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:80 Protocol:TCP HostIP:}] Env:<nil> Resources:{Limits:map[cpu:100m] Requests:map[]} VolumeMounts:[{Name:default-token-7sezo ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount}] LivenessProbe:<_>(0xc209d2da70){Handler:{Exec:<nil> HTTPGet:<*>(0xc20898c190){Path:/index.html Port:8080 Host: Scheme:HTTP} TCPSocket:<nil>} InitialDelaySeconds:30 TimeoutSeconds:1} ReadinessProbe:<nil> Lifecycle:<nil> TerminationMessagePath:/dev/termination-log ImagePullPolicy:IfNotPresent SecurityContext:<nil>}] RestartPolicy:Always TerminationGracePeriodSeconds:<nil> ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:<nil> ServiceAccountName: NodeName:gke-gke-upgrade-13e30713-node-xg2o HostNetwork:false ImagePullSecrets:<nil>}': may not update fields other than container.image

(#11343 improves this error message)

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 31 (17 by maintainers)

Most upvoted comments

@n1603 looks like the systemd unit files include the ServiceAccount admission controller without specifying the needed signing key. Not sure what to do about that, since those files don’t really have a setup script that can create that key…

To get your setup working, you can do the same thing local-up-cluster.sh is doing:

  1. Generate a signing key:

    openssl genrsa -out /tmp/serviceaccount.key 2048
    
  2. Update /etc/kubernetes/apiserver:

    KUBE_API_ARGS="--service_account_key_file=/tmp/serviceaccount.key"
    
  3. Update /etc/kubernetes/controller-manager:

    KUBE_CONTROLLER_MANAGER_ARGS="--service_account_private_key_file=/tmp/serviceaccount.key"