meshery: Unable to connect to EKS clusters using out-of-cluster docker Meshery server

Current Behavior

Tried two ways of adding clusters but both failed. Method 1: Follow the https://docs.meshery.io/installation/platforms/eks

$ mesheryctl system start -p docker
Updating Meshery now...
Pulling meshery              ... done
Pulling meshery-app-mesh     ... done
Pulling meshery-cilium       ... done
Pulling meshery-consul       ... done
Pulling meshery-istio        ... done
Pulling meshery-kuma         ... done
Pulling meshery-linkerd      ... done
Pulling meshery-nginx-sm     ... done
Pulling meshery-nsm          ... done
Pulling meshery-osm          ... done
Pulling meshery-traefik-mesh ... done
Pulling watchtower           ... done
The endpoint address will be changed to localhost. Are you sure you want to continue? [y/n]? y
Starting Meshery...
Creating meshery_meshery-consul_1       ... done
Creating meshery_meshery-istio_1        ... done
Creating meshery_meshery-nginx-sm_1     ... done
Creating meshery_meshery-app-mesh_1     ... done
Creating meshery_meshery-traefik-mesh_1 ... done
Creating meshery_meshery-linkerd_1      ... done
Creating meshery_meshery-kuma_1         ... done
Creating meshery_meshery_1              ... done
Creating meshery_meshery-cilium_1       ... done
Creating meshery_meshery-osm_1          ... done
Creating meshery_watchtower_1           ... done
Creating meshery_meshery-nsm_1          ... done
Opening Meshery (http://localhost:32093) in browser.
Failed to open Meshery in your browser, please point your browser to http://localhost:32093 to access Meshery.

Then:

$ mesheryctl system config eks
aws-cli/2.10.0 Python/3.9.11 Linux/5.15.0-1027-gcp exe/x86_64.ubuntu.20 prompt/off
Configuring Meshery to access EKS...
Please enter the AWS region name:
us-east-1
Please enter the AWS cluster name:
mesh-dev
Updated context arn:aws:eks:us-east-1:357476491302:cluster/mesh-dev in /home/gen_yang/.meshery/kubeconfig.yaml
Error getting context: Authentication token not found. Login with `mesheryctl system login` or supply a valid user token using the --token (or -t) flag.

So as the error message prompt:

$ mesheryctl system login
✔ Meshery
Initiating login...
authentication failed: Unable to reach Meshery server at http://localhost:32093. Verify your environment's readiness for a Meshery deployment by running `mesheryctl system check`.

So I check my cluster state:

$mesheryctl system check

Docker 
--------------
✓ Docker is running
✓ docker-compose is available

Kubernetes API 
--------------
✓ can initialize Kubernetes client
✓ can query the Kubernetes API

Kubernetes Version 
--------------
✓ running the minimum Kubernetes version
✓ running the minimum kubectl version

Meshery Version 
--------------
✓ Meshery Server is up-to-date (stable-v0.6.53)
✓ CLI is up-to-date (stable-v0.6.53)

Meshery Components 
--------------
!! Authentication token not found. Please supply a valid user token. Login with `mesheryctl system login`

That’s a loop, then I tried manually provide the token file to mesheryctl system config eks I download the token from the UI, it looks like this

{
    "meshery-provider":"Meshery",
    "token":"token"
}

Then I supply the token to the command, unluckily, it panicked

$ mesheryctl system config eks -t ./auth.json 
aws-cli/2.10.0 Python/3.9.11 Linux/5.15.0-1027-gcp exe/x86_64.ubuntu.20 prompt/off
Configuring Meshery to access EKS...
Please enter the AWS region name:
us-east-1
Please enter the AWS cluster name:
mesh-dev
Updated context arn:aws:eks:us-east-1:357476491302:cluster/mesh-dev in /home/gen_yang/.meshery/kubeconfig.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x22a0776]

goroutine 1 [running]:
github.com/layer5io/meshery/mesheryctl/internal/cli/root/system.setToken()
        /Users/runner/work/meshery/meshery/mesheryctl/internal/cli/root/system/config.go:392 +0xb6
github.com/layer5io/meshery/mesheryctl/internal/cli/root/system.glob..func12(0x43a55c0?, {0xc0009c4ae0?, 0x2?, 0x2?})
        /Users/runner/work/meshery/meshery/mesheryctl/internal/cli/root/system/config.go:252 +0x7af
github.com/spf13/cobra.(*Command).execute(0x43a55c0, {0xc0009c4ac0, 0x2, 0x2})
        /Users/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0x43a0540)
        /Users/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/layer5io/meshery/mesheryctl/internal/cli/root.Execute()
        /Users/runner/work/meshery/meshery/mesheryctl/internal/cli/root/root.go:81 +0x49
main.main()
        /Users/runner/work/meshery/meshery/mesheryctl/cmd/mesheryctl/main.go:24 +0x19

Method2 Since the cli can’t help me out, I tried to manually add the cluster on the Meshery UI by supplying the Kubeconfig file. It failed again with no error messages in the frontend. I check the Meshery server container logs and it shows:

time="2023-02-23T07:39:10Z" level=error msg="Kubernetes config is not initialized with Meshery" app=meshery code=2000 probable-cause="Kubernetes config is not accessible to meshery or not valid" severity=2 short-description="No valid kubernetes config found. Make sure to pass contextIDs in query parameters." suggested-remediation="Upload your kubernetes config via the settings dashboard. If uploaded, wait for a minute for it to get initialized"
time="2023-02-23T07:39:11Z" level=info msg="Cluster Resources subscription stopped" app=meshery
time="2023-02-23T07:39:11Z" level=info msg="Initializing MeshModelSummary subscription" app=meshery
time="2023-02-23T07:39:12Z" level=info msg="attempting to fetch kubernetes contexts from cloud for Meshery instance: ae33241a-d586-4fae-a47b-967cc47f55ff"
time="2023-02-23T07:39:12Z" level=info msg="kubernetes contexts successfully retrieved from remote provider"
time="2023-02-23T07:39:12Z" level=warning msg="failed to get kubernetes contexts"
time="2023-02-23T07:39:12Z" level=warning msg="failed to generate in cluster context: "
time="2023-02-23T07:39:12Z" level=warning msg="failed to load kubernetes contexts: "
time="2023-02-23T07:39:12Z" level=error msg="Kubernetes config is not initialized with Meshery" app=meshery code=2000 probable-cause="Kubernetes config is not accessible to meshery or not valid" severity=2 short-description="No valid kubernetes config found. Make sure to pass contextIDs in query parameters." suggested-remediation="Upload your kubernetes config via the settings dashboard. If uploaded, wait for a minute for it to get initialized"
time="2023-02-23T07:39:18Z" level=info msg="Generated context: arn:aws:eks:us-east-1:357476491302:cluster/mesh-dev\n"
time="2023-02-23T07:39:18Z" level=warning msg="Skipping context: Reason => Get \"https://D8B17DA21C8599E6268A377FD90AE201.gr7.us-east-1.eks.amazonaws.com/livez?timeout=1s\": getting credentials: exec: executable aws not found\n\nIt looks like you are trying to use a client-go credential plugin that is not installed.\n\nTo learn more about this feature, consult the documentation available at:\n      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins"

It seems that the server container can’t find aws binary in $PATH, which is related to the closed issues below https://github.com/meshery/meshery/issues/4184 https://github.com/meshery/meshery/pull/4190

Expected Behavior

Connect to multiple EKS clusters

Screenshots/Logs

Environment

• Host OS: Ubuntu focal 20.04
• Platform: Docker
• Meshery Server Version: stable-v
• Meshery Client Version: stable-v
• install

---

Contributor Guides and Handbook

• 🛠 Meshery Build & Release Strategy
• 📚 Instructions for contributing to documentation
  • Meshery documentation site and source
• 🎨 Wireframes and designs for Meshery UI in Figma
• 🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack