spinnaker: Deploy failed: error: unable to recognize "STDIN": Unauthorized

Issue Summary:

Deploy (Manifest) step return Deploy failed: error: unable to recognize "STDIN": Unauthorized Upon checking cloud driver logs, i got server does not have resource type 'namespace' and this came from all 4 clusters. This error randomly appears and disappears i.e. I have tried deploying a k8 manifest successfully in one execution and fail in another which I conclude to not be a permissions issue with GKE / k8

Cloud Provider(s):

Kubernetes v2

Environment:

Spinnaker is deployed on GKE and is configured to deploy into other GKE clusters in both the same GCP project and different GCP projects

Feature Area (if this issue is UI/UX related, please tag @spinnaker/ui-ux-team):

Description:

RBAC and k8 service account is not configured but for every k8 account configured on spinnaker, each of them uses a GCP service account with Kubernetes Engine Admin role for the cluster they are interacting with.

Steps to Reproduce:

create a Deploy (manifest step) with this manifest

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations: {}
  name: verycoolapp
spec:
  selector:
    matchLabels:
      app: foo
      tier: foo
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: foo
        extraLabel: extraVal
        tier: foo
      name: verycoolapp
    spec:
      containers:
        - image: 'yeasy/simple-web:latest'
          imagePullPolicy: IfNotPresent
          name: foo
          ports:
            - containerPort: 80
              name: foo
              protocol: TCP
          volumeMounts:
            - mountPath: /tmp
              name: tmp
      volumes:
        - emptyDir: {}
          name: tmp

Additional Details:

name: default
version: 1.12.6
providers:
  appengine:
    enabled: false
    accounts: []
  aws:
    enabled: false
    accounts: []
    bakeryDefaults:
      baseImages: []
    defaultKeyPairTemplate: '{{name}}-keypair'
    defaultRegions:
    - name: us-west-2
    defaults:
      iamRole: BaseIAMRole
  ecs:
    enabled: false
    accounts: []
  azure:
    enabled: false
    accounts: []
    bakeryDefaults:
      templateFile: azure-linux.json
      baseImages: []
  dcos:
    enabled: false
    accounts: []
    clusters: []
  dockerRegistry:
    enabled: true
    accounts:
    - name: docker-registry
      requiredGroupMembership: []
      providerVersion: V1
      permissions: {}
      address: https://index.docker.io
      username: <redacted>
      password: <redacted>
      email: <redacted>
      cacheIntervalSeconds: 30
      clientTimeoutMillis: 60000
      cacheThreads: 1
      paginateSize: 100
      sortTagsByDate: false
      trackDigests: false
      insecureRegistry: false
      repositories:
      - <redacted>
    - name: google-registry
      requiredGroupMembership: []
      providerVersion: V1
      permissions: {}
      address: https://asia.gcr.io
      username: _json_key
      email: fake.email@spinnaker.io
      cacheIntervalSeconds: 30
      clientTimeoutMillis: 60000
      cacheThreads: 1
      paginateSize: 100
      sortTagsByDate: false
      trackDigests: false
      insecureRegistry: false
      repositories: []
      passwordFile: /home/ubuntu/.gcp/spinnaker.json
    primaryAccount: docker-registry
  google:
    enabled: true
    accounts:
    - name: mygce
      requiredGroupMembership: []
      providerVersion: V1
      permissions: {}
      project: <redacted>
      jsonPath: /home/ubuntu/.gcp/spinnaker.json
      alphaListed: false
      imageProjects: []
      consul:
        enabled: false
        agentEndpoint: localhost
        agentPort: 8500
        datacenters: []
    primaryAccount: mygce
    bakeryDefaults:
      templateFile: gce.json
      baseImages: []
      zone: us-central1-f
      network: default
      useInternalIp: false
  kubernetes:
    enabled: true
    accounts:
    - name: cluster1
      requiredGroupMembership: []
      providerVersion: V2
      permissions: {}
      dockerRegistries: []
      context: context1
      configureImagePullSecrets: false
      cacheThreads: 1
      namespaces: []
      omitNamespaces: []
      kinds: []
      omitKinds: []
      customResources: []
      cachingPolicies: []
      kubeconfigFile: /home/ubuntu/.kube/config
      checkPermissionsOnStartup: true
      liveManifestCalls: true
      oauthScopes: []
      oAuthScopes: []
      onlySpinnakerManaged: true
    - name: cluster2
      requiredGroupMembership: []
      providerVersion: V2
      permissions: {}
      dockerRegistries: []
      context: sw-staging
      configureImagePullSecrets: false
      cacheThreads: 1
      namespaces: []
      omitNamespaces: []
      kinds: []
      omitKinds: []
      customResources: []
      cachingPolicies: []
      kubeconfigFile: /home/ubuntu/.kube/config
      checkPermissionsOnStartup: true
      liveManifestCalls: true
      oauthScopes: []
      oAuthScopes: []
      onlySpinnakerManaged: true
    - name: cluster3
      requiredGroupMembership: []
      providerVersion: V2
      permissions: {}
      dockerRegistries: []
      context: aux
      configureImagePullSecrets: false
      cacheThreads: 1
      namespaces: []
      omitNamespaces: []
      kinds: []
      omitKinds: []
      customResources: []
      cachingPolicies: []
      kubeconfigFile: /home/ubuntu/.kube/config
      checkPermissionsOnStartup: true
      liveManifestCalls: true
      oauthScopes: []
      oAuthScopes: []
      onlySpinnakerManaged: true
    - name: cluster4
      requiredGroupMembership: []
      providerVersion: V2
      permissions: {}
      dockerRegistries: []
      context: spinnaker
      configureImagePullSecrets: false
      cacheThreads: 1
      namespaces: []
      omitNamespaces: []
      kinds: []
      omitKinds: []
      customResources: []
      cachingPolicies: []
      kubeconfigFile: /home/ubuntu/.kube/config
      checkPermissionsOnStartup: true
      liveManifestCalls: true
      oauthScopes: []
      oAuthScopes: []
      onlySpinnakerManaged: true
    primaryAccount: cluster1
  openstack:
    enabled: false
    accounts: []
    bakeryDefaults:
      baseImages: []
  oracle:
    enabled: false
    accounts: []
    bakeryDefaults:
      templateFile: oci.json
      baseImages: []
  cloudfoundry:
    enabled: false
    accounts: []
deploymentEnvironment:
  size: SMALL
  type: Distributed
  accountName: cluster1
  updateVersions: true
  consul:
    enabled: false
  vault:
    enabled: false
  customSizing: {}
  sidecars: {}
  initContainers: {}
  hostAliases: {}
  nodeSelectors: {}
  gitConfig:
    upstreamUser: spinnaker
  haServices:
    clouddriver:
      enabled: false
      disableClouddriverRoDeck: false
    echo:
      enabled: false
persistentStorage:
  persistentStoreType: gcs
  azs: {}
  gcs:
    jsonPath: /home/ubuntu/.gcp/spinnaker.json
    project: <redacted>
    bucket: <redacted>
    rootFolder: front50
    bucketLocation: <redacted>
  redis: {}
  s3:
    rootFolder: front50
  oracle: {}
features:
  auth: false
  fiat: false
  chaos: false
  entityTags: false
  jobs: false
  artifacts: true
metricStores:
  datadog:
    enabled: false
    tags: []
  prometheus:
    enabled: false
    add_source_metalabels: true
  stackdriver:
    enabled: false
  period: 30
  enabled: false
notifications:
  slack:
    enabled: false
  twilio:
    enabled: false
    baseUrl: https://api.twilio.com/
timezone: America/Los_Angeles
ci:
  jenkins:
    enabled: false
    masters: []
  travis:
    enabled: false
    masters: []
  wercker:
    enabled: false
    masters: []
security:
  apiSecurity:
    ssl:
      enabled: true
      keyAlias: gatestore
      keyStore: /home/ubuntu/.keystore/gatestore.jks
      keyStoreType: jks
      keyStorePassword: <redacted>
    overrideBaseUrl: <redacted>
  uiSecurity:
    ssl:
      enabled: true
      sslCertificateFile: <redacted>
      sslCertificateKeyFile: <redacted>
      sslCertificatePassphrase: <redacted>
    overrideBaseUrl: <redacted>
  authn:
    oauth2:
      enabled: true
      client:
        clientId: <redacted>
        clientSecret: <redacted>
        accessTokenUri: https://www.googleapis.com/oauth2/v4/token
        userAuthorizationUri: https://accounts.google.com/o/oauth2/v2/auth
        scope: profile email
      userInfoRequirements:
        hd: <redacted>
      resource:
        userInfoUri: https://www.googleapis.com/oauth2/v3/userinfo
      userInfoMapping:
        email: email
        firstName: given_name
        lastName: family_name
      provider: GOOGLE
    saml:
      enabled: false
      userAttributeMapping: {}
    ldap:
      enabled: false
    x509:
      enabled: false
    iap:
      enabled: false
    enabled: true
  authz:
    groupMembership:
      service: EXTERNAL
      google:
        roleProviderType: GOOGLE
      github:
        roleProviderType: GITHUB
      file:
        roleProviderType: FILE
      ldap:
        roleProviderType: LDAP
    enabled: false
artifacts:
  bitbucket:
    enabled: false
    accounts: []
  gcs:
    enabled: true
    accounts:
    - name: gcs-artifact-account
      jsonPath: /home/ubuntu/.gcp/spinnaker.json
  oracle:
    enabled: false
    accounts: []
  github:
    enabled: true
    accounts:
    - name: github-artifact-account
      tokenFile: /home/ubuntu/.github/token
  gitlab:
    enabled: false
    accounts: []
  http:
    enabled: false
    accounts: []
  helm:
    enabled: false
    accounts: []
  s3:
    enabled: false
    accounts: []
  maven:
    enabled: false
    accounts: []
  templates: []
pubsub:
  enabled: true
  google:
    enabled: true
    pubsubType: GOOGLE
    subscriptions:
    - name: helm-artifact-update
      project: <redacted>
      subscriptionName: <redacted>
      jsonPath: /home/ubuntu/.gcp/spinnaker.json
      ackDeadlineSeconds: 10
      messageFormat: GCS
    publishers: []
canary:
  enabled: false
  serviceIntegrations:
  - name: google
    enabled: false
    accounts: []
    gcsEnabled: false
    stackdriverEnabled: false
  - name: prometheus
    enabled: false
    accounts: []
  - name: datadog
    enabled: false
    accounts: []
  - name: signalfx
    enabled: false
    accounts: []
  - name: aws
    enabled: false
    accounts: []
    s3Enabled: false
  reduxLoggerEnabled: true
  defaultJudge: NetflixACAJudge-v1.0
  stagesEnabled: true
  templatesEnabled: true
  showAllConfigsEnabled: true
webhook:
  trust:
    enabled: false

cloud driver logs kubernetes:cluster<1-4> reported same error as well

2019-03-28 09:14:42.755  WARN 1 --- [utionAction-277] c.n.s.c.cache.LoggingInstrument
ation     : kubernetes:cluster1/KubernetesNamespaceCachingAgent[1/1] compl
eted with one or more failures

com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor$NoResourceT
ypeException: error: the server doesn't have a resource type "namespace"

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

This issue is tagged as ‘to-be-closed’ and hasn’t been updated in 45 days, so we are closing it. You can always reopen this issue if needed.

+2
spinnakerbot on Feb 6, 2020

even with a v2 account, I’m running into this (on 1.14.9). Please don’t kill it, oh spinnakerbot

edit: I remade my service account and all is well now:

CONTEXT=$(kubectl config current-context)

kubectl apply --context $CONTEXT \
    -f https://spinnaker.io/downloads/kubernetes/service-account.yml

TOKEN=$(kubectl get secret --context $CONTEXT \
   $(kubectl get serviceaccount spinnaker-service-account \
       --context $CONTEXT \
       -n spinnaker \
       -o jsonpath='{.secrets[0].name}') \
   -n spinnaker \
   -o jsonpath='{.data.token}' | base64 --decode)

kubectl config set-credentials ${CONTEXT}-token-user --token $TOKEN

kubectl config set-context $CONTEXT --user ${CONTEXT}-token-user

hal config provider kubernetes enable

CONTEXT=$(kubectl config current-context)

hal config provider kubernetes account add my-k8s-v2-account \
    --provider-version v2 \
    --context $CONTEXT
+2
adamdavis40208 on Jul 16, 2019
Read more comments on GitHub
← spinnaker: Deck will not start with ssl enabled
spinnaker: Deployment fails when Azure Container Registry is used →