provider-terraform: AWS TF Provider Fails to Use Provided Secrets

What happened?

I’m trying to sync a terraform module using this provider + argocd. While trying to extrapolate some of the patterns from examples like the aws ini example I’m seeing errors like:

cannot diff (i.e. plan) Terraform configuration: error configuring terraform aws provider:
no valid credential sources for terraform aws provider found.

How can we reproduce it?

I’m running all of this on a KinD cluster.

1. KinD local cluster config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane

  image: kindest/node:v1.23.0@sha256:49824ab1727c04e56a21a5d8372a402fcd32ea51ac96a2706a12af38934f81ac

  extraPortMappings:
  - containerPort: 8443
    hostPort: 8443
  - containerPort: 30346
    hostPort: 9443

Spinning it up via:

kind create cluster --name $(cluster-name) --config kind-local-cluster-cfg.yaml

2. Installing Crossplane (namespace: crossplane-system)

kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane
ls /usr/local/bin/kubectl-crossplane || (curl -sL \
    https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh)

As of writing this issue, crossplane itself is helm.sh/chart: crossplane-1.10.2

I’m then configuring installing the TF-Xplane provider via the following manifest:

---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-terraform
  namespace: crossplane-system
spec:
  package: xpkg.upbound.io/upbound/provider-terraform:v0.3.0
  controllerConfigRef:
    name: provider-terraform-controller-config

---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: provider-terraform-controller-config
  namespace: crossplane-system
  labels:
    app: crossplane-provider-terraform
spec:
  image: xpkg.upbound.io/upbound/provider-terraform:v0.3.0
  args: ["-d"]

3. Setting Up Credentials (namespace: foo)

On the same KinD cluster I then have a separate namespace, say foo.

In it, I have a Secret with the following contents base64 encoded:

[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
aws_session_token = $AWS_SESSION_TOKEN
aws_security_token = $AWS_SECURITY_TOKEN

i.e.: I use env-vars to replace the values above, and the secret is created via something akin to:

cat local/aws-creds-template.ini | envsubst > real-aws-creds.ini

kubectl create secret generic my-aws-creds \
    --namespace foo \
    --from-file=credentials=./real-aws-creds.ini

The same is with some git credentials:

example template file

https://${username}:${corp_git_token}@git.my.corp.com

And how Im generating the secret

cat local/git-creds-template.txt | envsubst > real-git-creds.txt

kubectl create secret generic my-git-creds \
    --namespace foo \
    --from-file=b64data=./real-git-creds.txt

4. Creating XRDs and Testing Provider (namespace: foo)

In the same namespace I have the following configs (yes I know that these are cluster-wide but I’m setting things up to experiment w/ OPA in somewhere down the line):

ProviderConfig

apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: local-kind-tf-provider-config
spec:
  configuration: |
    terraform {
      backend "kubernetes" {
        secret_suffix     = "tf-providerconfig-aws"
        namespace         = "foo"
        in_cluster_config = true
      }

      required_providers {
        aws = {
          source = "hashicorp/aws"
          version = "3.74.2"
        }
      }
    }

    provider "aws" {
      profile                 = "default"
      region                  = "us-east-1"
      shared_credential_files = ["aws-config.ini"]
      shared_credentials_file = "aws-config.ini"
    }
  credentials:
    - filename: aws-config.ini
      secretRef:
        key: credentials
        name: my-aws-creds
        namespace: foo
      source: Secret
    - filename: .git-credentials
      secretRef:
        key: b64data
        name: my-git-creds
        namespace: foo
      source: Secret

And my Workspace

apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  name: local-xplane-argocd-terraform-xrm-s3-example-bucket-workspace
spec:
  deletionPolicy: Delete
  forProvider:
    entrypoint: ''
    module: >-
      git::https://git.my.corp.com/mee/crossplane-argocd-playground.git//./examples/terraform-xrm-s3/src?ref=HEAD
    source: Remote
    vars:
      - key: bucket_name
        value: my-test-bucket
      - key: launch_source
        value: my-laptop
      - key: owner
        value: mee
      - key: region
        value: us-east-1
  providerConfigRef:
    name: local-kind-tf-provider-config
  writeConnectionSecretToRef:
    name: >-
      random-s3-details
    namespace: foo

I’m going to omit the contents of ./examples/terraform-xrm-s3 in the repository that I’m syncing with for now because it does not seem to be the problem (i.e.: I removed the git credentials in the ProviderConfig at some point and the connection w/ our private github instance indeed failed). But the contents is just a module that wraps the aws_s3_bucket resource w/ some tags.

What environment did it happen in?

  • Universal Crossplane Version: crossplane-1.10.2
  • Provider Version: 0.3.0
  • Running on an X86 mac machine w/ Docker Desktop 4.15
  • k8s:
» kubectl version -o=yaml
clientVersion:
  buildDate: "2022-09-21T14:33:49Z"
  compiler: gc
  gitCommit: 5835544ca568b757a8ecae5c153f317e5736700e
  gitTreeState: clean
  gitVersion: v1.25.2
  goVersion: go1.19.1
  major: "1"
  minor: "25"
  platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2021-12-08T00:29:07Z"
  compiler: gc
  gitCommit: ab69524f795c42094a6630298ff53f3c3ebab7f4
  gitTreeState: clean
  gitVersion: v1.23.0
  goVersion: go1.17.3
  major: "1"
  minor: "23"
  platform: linux/amd64

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 30 (16 by maintainers)

Most upvoted comments

@chr0n1x have you tried exec’ing into the provider-terraform pod and examining the content of the /tf/UUID_STRING directory to make sure that the aws-creds.ini file is being populated correctly? That would be the first thing I would check.

@chr0n1x glad you got something working! I think this will be directly usable in #70 just with assume_role instead of the token/keys.