kubeflow: Could not determinte it's EKS cluster MissingRegion: could not find region configuration

/kind bug

What steps did you take and what happened: Trying to install Kubeflow in AWS EKS cluster using kfctl apply -f kfctl_aws.yaml

What did you expect to happen: Kubeflow is successfully installed

Anything else you would like to add: Running this inside docker container

Environment:

Kubeflow version: 1.0.1:
kfctl version: (use kfctl version): 1.0.1

I am getting an error:

kfctl apply --verbose --file kfctl_aws.1.0.1.yaml time=“2020-03-11T22:30:20Z” level=info msg=“Running eksctl version …” filename=“utils/awsutil.go:71” time=“2020-03-11T22:30:20Z” level=info msg=“eksctl version: 0.14.0” filename=“utils/awsutil.go:88” time=“2020-03-11T22:30:20Z” level=info msg=“\n****************************************************************\nNotice anonymous usage reporting enabled using spartakus\nTo disable it\nIf you have already deployed it run the following commands:\n cd $(pwd)\n kubectl -n ${K8S_NAMESPACE} delete deploy -l app=spartakus\n\nFor more info: https://www.kubeflow.org/docs/other-guides/usage-reporting/\n****************************************************************\n\n” filename=“coordinator/coordinator.go:120” time=“2020-03-11T22:30:20Z” level=info msg=“.cache/manifests exists; not resyncing " filename=“kfconfig/types.go:468” time=“2020-03-11T22:30:20Z” level=info msg=“Folder aws_config exists, skip aws.Generate” filename=“aws/aws.go:438” time=“2020-03-11T22:30:21Z” level=info msg=“folder kustomize exists, skip kustomize.Generate” filename=“kustomize/kustomize.go:372” time=“2020-03-11T22:30:21Z” level=info msg=”.cache/manifests exists; not resyncing " filename=“kfconfig/types.go:468” time=“2020-03-11T22:30:21Z” level=info msg=“Caller ARN Info: {\n Account: "277749501672",\n Arn: "arn:aws:sts::111111111:assumed-role/service-deploy/service-deploy",\n UserId: "AAAAAAAA-deploy"\n}” filename=“utils/awsutil.go:41” time=“2020-03-11T22:30:21Z” level=info msg=“ManagedCluster set defaulting to false” filename=“aws/aws.go:651” time=“2020-03-11T22:30:21Z” level=info msg=“EnablePodIamPolicy set defaulting to false” filename=“aws/aws.go:656” time=“2020-03-11T22:30:21Z” level=info msg=“EnableNodeGroupLog set defaulting to false” filename=“aws/aws.go:661” time=“2020-03-11T22:30:21Z” level=info msg="You already have cluster setup. Skip creating new eks cluster. " filename=“aws/eks.go:95” Error: failed to apply: (kubeflow.error): Code 500 with message: coordinator Apply failed for aws: (kubeflow.error): Code 400 with message: Could not determinte it’s EKS cluster MissingRegion: could not find region configuration

This is my cluster config:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: build
  region: us-west-2
  version: "1.14"
nodeGroups:
- desiredCapacity: 2
  instanceType: m5.xlarge
  maxSize: 3
  minSize: 0
  name: cpu-nodegroup
  volumeSize: 30

Also the IAM user the deployment runs under has the region configured: export AWS_DEFAULT_REGION=us-west-2

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 27 (14 by maintainers)

Most upvoted comments

Hi all, a quick workaround for this issue is:

export AWS_SDK_LOAD_CONFIG=1

This can make session read configuration information from config file (~/.aws/config).

PatrickXYS on Mar 17, 2020

Set the region of your aws cli and it will work:

aws configure set default.region us-west-2

But this is a bug because kfctl completely ignores the region set in the .yaml file and just uses the aws cli default.

fsaina on Mar 15, 2020

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: manifest
  region: us-west-2
  version: "1.14"
nodeGroups:
- desiredCapacity: 2
  instanceType: m5.xlarge
  maxSize: 3
  minSize: 0
  name: cpu-nodegroup
  volumeSize: 30

Hi all, please don’t be confused about this. This was used to created a kubeflow with a new EKS cluster. Now, the recommend path is to deploy kubeflow on existing EKS cluster which means this file is not helpful.

Error: failed to apply: (kubeflow.error): Code 500 with message: coordinator Apply failed for aws: (kubeflow.error): Code 400 with message: Could not determinte it's EKS cluster MissingRegion: could not find region configuration

This indicates that is failed here.

https://github.com/kubeflow/kfctl/blob/1fe560ef9bdb3e8aac135f0e7ff45e7d097bdf17/pkg/kfapp/aws/aws.go#L710-L716

it uses default resolver to make a call to EKS https://github.com/kubeflow/kfctl/blob/1fe560ef9bdb3e8aac135f0e7ff45e7d097bdf17/pkg/kfapp/aws/eks.go#L49-L65

Please make sure you have AWS_REGION environment set in your environment to mitigate this issue temporarily . We will file a PR improvement to use plugin’s region user filled to create a SDK session, or resolve profile provider properly.

https://docs.aws.amazon.com/es_es/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-the-region

Jeffwan on Mar 16, 2020

@apryiomka Since according to the AWS SDK Go doc, we should set the region with

export AWS_REGION=us-west-2

For reference:

But this is just for workaround, I’ll fix it in the PR and you can see it in the next patch revision of kfctl.

PatrickXYS on Mar 17, 2020

Hi all, a quick workaround for this issue is:
export AWS_SDK_LOAD_CONFIG=1 
This can make session read configuration information from config file (~/.aws/config).

this works for me, thanks!

majorinche on Apr 15, 2020