provider-terraform: AWS TF Provider Fails to Use Provided Secrets
What happened?
I’m trying to sync a terraform module using this provider + argocd. While trying to extrapolate some of the patterns from examples like the aws ini example I’m seeing errors like:
cannot diff (i.e. plan) Terraform configuration: error configuring terraform aws provider:
no valid credential sources for terraform aws provider found.
How can we reproduce it?
I’m running all of this on a KinD cluster.
1. KinD local cluster config:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.23.0@sha256:49824ab1727c04e56a21a5d8372a402fcd32ea51ac96a2706a12af38934f81ac
extraPortMappings:
- containerPort: 8443
hostPort: 8443
- containerPort: 30346
hostPort: 9443
Spinning it up via:
kind create cluster --name $(cluster-name) --config kind-local-cluster-cfg.yaml
2. Installing Crossplane (namespace: crossplane-system)
kubectl create namespace crossplane-system
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system crossplane-stable/crossplane
ls /usr/local/bin/kubectl-crossplane || (curl -sL \
https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | sh)
As of writing this issue, crossplane itself is helm.sh/chart: crossplane-1.10.2
I’m then configuring installing the TF-Xplane provider via the following manifest:
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-terraform
namespace: crossplane-system
spec:
package: xpkg.upbound.io/upbound/provider-terraform:v0.3.0
controllerConfigRef:
name: provider-terraform-controller-config
---
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
name: provider-terraform-controller-config
namespace: crossplane-system
labels:
app: crossplane-provider-terraform
spec:
image: xpkg.upbound.io/upbound/provider-terraform:v0.3.0
args: ["-d"]
3. Setting Up Credentials (namespace: foo)
On the same KinD cluster I then have a separate namespace, say foo.
In it, I have a Secret with the following contents base64 encoded:
[default]
aws_access_key_id = $AWS_ACCESS_KEY_ID
aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
aws_session_token = $AWS_SESSION_TOKEN
aws_security_token = $AWS_SECURITY_TOKEN
i.e.: I use env-vars to replace the values above, and the secret is created via something akin to:
cat local/aws-creds-template.ini | envsubst > real-aws-creds.ini
kubectl create secret generic my-aws-creds \
--namespace foo \
--from-file=credentials=./real-aws-creds.ini
The same is with some git credentials:
example template file
https://${username}:${corp_git_token}@git.my.corp.com
And how Im generating the secret
cat local/git-creds-template.txt | envsubst > real-git-creds.txt
kubectl create secret generic my-git-creds \
--namespace foo \
--from-file=b64data=./real-git-creds.txt
4. Creating XRDs and Testing Provider (namespace: foo)
In the same namespace I have the following configs (yes I know that these are cluster-wide but I’m setting things up to experiment w/ OPA in somewhere down the line):
ProviderConfig
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
name: local-kind-tf-provider-config
spec:
configuration: |
terraform {
backend "kubernetes" {
secret_suffix = "tf-providerconfig-aws"
namespace = "foo"
in_cluster_config = true
}
required_providers {
aws = {
source = "hashicorp/aws"
version = "3.74.2"
}
}
}
provider "aws" {
profile = "default"
region = "us-east-1"
shared_credential_files = ["aws-config.ini"]
shared_credentials_file = "aws-config.ini"
}
credentials:
- filename: aws-config.ini
secretRef:
key: credentials
name: my-aws-creds
namespace: foo
source: Secret
- filename: .git-credentials
secretRef:
key: b64data
name: my-git-creds
namespace: foo
source: Secret
And my Workspace
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
name: local-xplane-argocd-terraform-xrm-s3-example-bucket-workspace
spec:
deletionPolicy: Delete
forProvider:
entrypoint: ''
module: >-
git::https://git.my.corp.com/mee/crossplane-argocd-playground.git//./examples/terraform-xrm-s3/src?ref=HEAD
source: Remote
vars:
- key: bucket_name
value: my-test-bucket
- key: launch_source
value: my-laptop
- key: owner
value: mee
- key: region
value: us-east-1
providerConfigRef:
name: local-kind-tf-provider-config
writeConnectionSecretToRef:
name: >-
random-s3-details
namespace: foo
I’m going to omit the contents of ./examples/terraform-xrm-s3 in the repository that I’m syncing with for now because it does not seem to be the problem (i.e.: I removed the git credentials in the ProviderConfig at some point and the connection w/ our private github instance indeed failed). But the contents is just a module that wraps the aws_s3_bucket resource w/ some tags.
What environment did it happen in?
- Universal Crossplane Version: crossplane-1.10.2
- Provider Version: 0.3.0
- Running on an X86 mac machine w/ Docker Desktop 4.15
- k8s:
» kubectl version -o=yaml
clientVersion:
buildDate: "2022-09-21T14:33:49Z"
compiler: gc
gitCommit: 5835544ca568b757a8ecae5c153f317e5736700e
gitTreeState: clean
gitVersion: v1.25.2
goVersion: go1.19.1
major: "1"
minor: "25"
platform: darwin/amd64
kustomizeVersion: v4.5.7
serverVersion:
buildDate: "2021-12-08T00:29:07Z"
compiler: gc
gitCommit: ab69524f795c42094a6630298ff53f3c3ebab7f4
gitTreeState: clean
gitVersion: v1.23.0
goVersion: go1.17.3
major: "1"
minor: "23"
platform: linux/amd64
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 5
- Comments: 30 (16 by maintainers)
@chr0n1x have you tried exec’ing into the provider-terraform pod and examining the content of the /tf/UUID_STRING directory to make sure that the aws-creds.ini file is being populated correctly? That would be the first thing I would check.
@chr0n1x glad you got something working! I think this will be directly usable in #70 just with assume_role instead of the token/keys.